# General

This directory contains all third-party dependencies of TheDarkMod, both the game and other tools.
There are some rare exceptions when library is embedded and tightly integrated directly into the engine, but this approach is generally avoided.
The dependencies are fetched and built using [conan][1] package manager.

Note that aside from headers and libs of third-party dependency, you also need *glue* for the build system you are going to use.
These are config files in case of CMake and props files in case of msbuild/vcxproj.
Almost all the glue code is generated by conan as well.

There are two ways to use conan in TDM:

 1. **Detached**: Conan is used only to build third-party artefacts: headers, libs, and glue for CMake/vcxproj.
    These artefacts cover the main platforms and default settings, and they are committed to SVN.
    After that all developers can build TDM using them without conan involved (not even installed).
 2. **Linked**: Conan is used to build third-party libs in customized way: custom platforms, custom versions, custom settings.
    After that conan is used to configure CMake and generate projects/makefiles which can be used to build TDM.

## Directory structure

Here are the contents of this directory:

 * `custom/*` --- custom conan recipes for some libraries (in case stock recipe is not available or does not fit)
 * `artefacts/*` --- resulting aftefacts (headers, libs, glue) for TDM projects on main/supported platforms
 * `profiles/*` --- conan settings for building artefacts for main platforms
 * `1_export_custom.py` --- script to export custom recipes into active conan cache
 * `2_build_all.py` --- script to build standalone artefacts for main platforms
 * `packages.yaml` --- data file which lists all libraries, their versions and options, and some metainfo
 * `conanfile.py` --- the main file which explains to conan how to install TDM dependencies
 * `tdm_deploy.py` --- custom conan deployer which is used to generate standalone artefacts for main platforms
 * `use_artefacts.cmake` --- cmake glue which is included to use prebuilt artefacts on main platforms

All third-party binaries **must** be inside `artefacts` directory!
It is available on SVN but usually ignored on DVCS (including GitHub mirror).


# Preparation

## Install conan

First of all, make sure that you have latest Python 3 installed, e.g. from [download page][2].

Next, install conan if not yet installed.
Detailed instructions are provided [here][3], the easiest way is via pip:

    pip install conan

You might want to read some "Getting Started" conan docs to get brief understanding of what it is and how it works.

## Locate conan cache

Conan builds packages inside a *local cache*, which by default is located somewhere in `%USERPROFILE%/.conan` and is in fact system-global.
If you want to make the cache truly local, you can set environment variable:

    # set the path to a custom conan cache directory
    set CONAN_HOME=G:\tdm_conan

Note that if you do so, you should ensure that this environment variable takes effect for all the conan commands you run later.
Detailed explanation is available [here][4].

You can omit this step, then system-wide cache will be used for building libraries.

## Export custom recipes

Recipes for most libraries are taken directly from central conan repository (i.e. "conan-center").
Some libraries need special handling for TDM, so we manage recipes for them ourselves. They are located in `custom` directory.

Before actually building the libs, it is necessary to "export" these recipes into conan cache.
The easiest way to do that is to run the small helper script:

    python 1_export_custom.py

The script will display all the conan console commands it wants to run.
Note that you need to rerun this script whenever custom recipes are changed, otherwise conan won't know about the changes.


# Build libraries: detached

This section applies only if you want to build artefacts for main platforms,
and commit them to SVN for other developers to use without conan.

You need Windows 64-bit + MSVC and Linux 64-bit + GCC in order to generate everything.
You might want to delete the `artefacts` directory to get only fresh output.

Run the helper script:

    python 2_build_all.py

The script will display all the conan console commands it wants to run.
Notice two arguments which are special for the *detached* approach:

 * `-of artefacts/windows_64`: tells conan where to put glue for CMake/vcxproj
 * `-d tdm_deploy`: tells conan to run our custom deployer script that would copy headers and libs

Remember that you need to run the script once on Windows and once on Linux to generate everything.
Given that produced artefacts may differ slightly on different machines (no two .lib-s are exactly equal),
please commit new artefacts only for the libraries which you have actually changed, don't commit changes in libraries which you did not touch.


# Build libraries: linked

This section applies if you want to build artefacts for yourself only.
It allows customizing various settings, and can be used if you build for a special platform.

Let's look at one of the commands executed by `2_build_all.py`:

    conan install . -pr:b profiles/base_windows -pr profiles/os_windows -pr profiles/arch_64 -pr profiles/build_release -s thedarkmod/*:build_type=Release -of artefacts/windows_64 -d tdm_deploy -b missing

`conan install .` tells to prepare all libraries and glue for the conanfile located in current directory.
It will build third-party libs, but it will not build the game itself.
You can replace it with `conan build -o build_game=True`, then conan will additionally build the game for you!
Other TDM tools can be built this way too, see `conanfile.py` for respective options.

`-pr profiles/os_windows`, `-pr profiles/arch_64`, `-pr profiles/build_release` specify "host profile" to build libs for.
Each of them is just a pack of settings for the build.
You can read about settings in [5], and about profiles in [6].
In general, you can avoid passing any profiles to conan if you want to build only for yourself: then conan will use auto-detected default profile.

In the detached case the full profile is comprised of three orthogonal choices: Linux/Windows OS, 32-bit or 64-bit x86 architecture, Release/Debug/whatever type of build.
Note that we use special "debugfast" type of build for third-party libs.
In this special build type the libs are compiled with optimization, but C++ libs are compiled against MSVC debug runtime since otherwise they are incompatible with Debug TDM code.
On GCC there is no difference between debugfast and release types.

`-pr:b profiles/base_windows` specifies the settings for "build profile".
Build profile is usually important when cross-compiling (see [7]) and for build tools like pkgconf, ninja, msys, etc.
Technically we do cross-compile when we build 32-bit binaries on 64-bit OS.
If you don't specify any build profile, then conan takes the default profile which was autodetected on your machine (which should be OK).
Make sure that at least architecture and build type are specified in build profile!
And never have debug build type in build profile, or conan will build you debug build of ninja =)

`-s thedarkmod/*:build_type=Release` says that Release build is going to be used for TDM itself.
This does not affect how third-party libraries are built, but it does affect how glue is generated.
The glue is targeted as specific build type of TDM and will only apply to it.
This is important for Debug build, where `profiles/build_debugfast` technically sets build_type=Release, but we actually use it for Debug builds.

`-b missing` asks conan to build third-party libraries if the binaries are not available yet in conan cache or on remove server.
Without this argument, conan lists all the packages without binaries and stops there.

`-of artefacts/windows_64` says where to put glue code for the build.
For linked builds you should specify something like `-of mybuild`.
It is extremely advised to specify `-of`, because otherwise all the glue garbage will trash the current directory!

`-d tdm_deploy` is special deployer, only used for "detached" workflow.
Do not use it.


Now here are some examples of custom conan commands.

    conan build . -of temp -o build_game=True -o build_installer=True -o build_packager=True -b missing

This starts build of all 3 TDM projects with default profile.
In my case it tries to use dynamic runtime, which is not compatible with TDM projects, so everything fails.
Just set runtime back to static (don't forget to delete temp directory between installs):

    conan build . -of temp -s compiler.runtime=static -o build_game=True -o build_installer=True -o build_packager=True -b missing

This works fine for me and builds 3 executables in `temp\build_*\Release`.

Another example is using unoptimized Debug build of third-party libs (not recommended but possible):

    conan install . -pr:b profiles/base_windows -pr profiles/os_windows -pr profiles/arch_64 -pr profiles/build_debug -s thedarkmod/*:build_type=Debug -of temp

Here we reuse the profiles from detached workflow, but use build_debug profile for the third-party libs.
Note that base_windows is passed as build profile here, although maybe you can drop it.
This command will only show you which libs need to be built, add `-b missing` to actually build them.


# How to add new library

You can search for recipes of a library like this:

    conan search {libraryname}

It might be useful to add asterisk at the beginning and at the ending of the library name.

If you manage to find the library, then choose one of the printed references and add it to the list in `packages.yaml`.
After that you can do usual build and your library will get downloaded and built.

If you fail to find ready-to-use conan recipe, then you have to add a custom one.
Note that making a custom recipe most likely requires good conan skills!
You might want to search it on github with codenames `conan` and library name.
If you find someone else's recipe, you can add it to `custom` and it will probably work.
A lot of existing recipes were made for conan 1, and won't work with conan 2 that we use now.

Sometimes there is a recipe for a library, but it needs fine-tuning in order to be used in TDM.
In such case you should download the recipe, add it to `custom`, and edit there.
Make sure to clearly mark all your changes to the recipe, or/and save its original version nearby.

Otherwise you have to create recipe yourself, or ask someone else to do it on forums.
In principle, it is possible to build the library manually and add artefacts to `artefacts` directory.
But it would be really hard to cover all main platforms and generate glue properly for all build systems.


## How to update library

Look into `packages.yaml` and find the library reference there.
Now do `conan search` for a newer version of the library.

If you find one, change the reference in `conanfile.py` and rebuild.
Otherwise, you are out of luck.

If you want to update a library with custom recipe, it gets more complicated.
In many cases you can apply an existing recipe to the newer version: just add source code URL and sha256 checksum for the new version in `conandata.yml`.
If this is custom-modified recipe, then you can fetch a newer recipe from central repository and try to reapply the same TDM-specified hacks to it.



[1]: https://conan.io/
[2]: https://www.python.org/downloads/
[3]: https://docs.conan.io/2/installation.html
[4]: https://docs.conan.io/2/reference/environment.html#conan-home
[5]: https://docs.conan.io/2/reference/config_files/settings.html
[6]: https://docs.conan.io/2/reference/config_files/profiles.html
[7]: https://docs.conan.io/2/tutorial/consuming_packages/cross_building_with_conan.html#consuming-packages-cross-building-with-conan