Configuration

Introduction

pyqalx has various configurable options. These allow you to adjust the operation of the system based on the specific task you’re performing.

For the most part, these can be left as the default values. More detail on overriding defaults can be found in Custom config and overriding defaults below.

Default values

Global

These values are set on all configuration objects throughout the system.

pyqalx.config.defaults.TOKEN = None

API token

pyqalx.config.defaults.LOGGING_LEVEL = 'ERROR'

Level that we should log at by default.

pyqalx.config.defaults.LOG_FILE_PATH = '/opt/buildhome/.cache/qalx/log/.qalx.log'

The path where log files will be stored.

  • Windows: C:\Users\<User>\AppData\Local\agiletek\qalx\Logs\.qalx.log

  • Linux: /home/<user>/.cache/qalx/log/.qalx.log

  • Max ~/Library/Logs/qalx/.qalx.log

pyqalx.config.defaults.MSG_BLACKOUTSECONDS = 30

After this time in seconds the message will be returned to the queue if it has not been deleted.

Note

Workers automatically issue a heartbeat to the queue while the job is being processed to keep extending the blackout during processing

pyqalx.config.defaults.MAX_RETRY_500 = 2

How many times should pyqalx retry the request when receiving a 5XX error back from the API?

pyqalx.config.defaults.FACTORY_PACK_DIR = '/opt/buildhome/.local/share/qalx'

The directory where factory bot code will be packed and deployed from. Defaults to the following:

  • Windows: C:\Users\<User>\AppData\Local\agiletek\qalx

  • Linux: ~/.local/share/qalx

  • Mac: ~/Library/Application Support/qalx

pyqalx.config.defaults.DOWNLOAD_DIR = '/opt/buildhome/.local/share/qalx'

The directory where files from qalx will be downloaded by default. Defaults to the following:

  • Windows: C:\Users\<User>\AppData\Local\agiletek\qalx

  • Linux: ~/.local/share/qalx

  • Mac: ~/Library/Application Support/qalx

pyqalx.config.defaults.REGION = 'eu-west-2'

The region that will be used when connecting to databases or other remote services. Originally set via qalx configure. Change this to another valid region to use services in that region. This is only a valid option for User configs - not for Bot configs

Bots

These values are set whenever Sessions are configured in the context of a bot. These configuration options should be added to your ~/.bots config file

pyqalx.config.defaults.bots.WORKER_LOG_FILE_DIR = '/opt/buildhome/.cache/qalx/log'

Directory to save worker log files

  • Windows: C:\Users\<User>\AppData\Local\agiletek\qalx\Logs

  • Linux: /home/<user>/.cache/qalx/log

  • Max ~/Library/Logs/qalx

pyqalx.config.defaults.bots.SAVE_INTERMITTENT = False

Save the entity after every processing step?

pyqalx.config.defaults.bots.SKIP_FINAL_SAVE = False

Skip saving the entity back to qalx after post-processing.

pyqalx.config.defaults.bots.KILL_AFTER = None

The number of attempts to get a message from an empty queue before exiting. When set to None the worker will keep asking forever.

pyqalx.config.defaults.bots.Q_WAITTIMESECONDS = 20

The minimum time in seconds to wait before asking for a message from an empty queue

pyqalx.config.defaults.bots.Q_MSGBATCHSIZE = 1

The number of messages to get in each request

pyqalx.config.defaults.bots.Q_WAITTIMEMAXSPREADING = 12

The maximum spreading factor for a wait. The maximum wait time will be Q_WAITTIMEMAXSPREADING*Q_WAITTIMESECONDS.

pyqalx.config.defaults.bots.MSG_WAITTIMESECONDS = 20

The time in seconds the bot will wait for a message to appear in the queue

Custom config and overriding defaults

All Sessions have a config attribute which returns an object that behaves exactly like a python dict. There are four main ways to customise the configuration:

.ini files

pyqalx looks for a .qalx and a .bots file in the home directory of the user. These files follow the INI file format and can be used to overwrite or add new configurations. This also supports multiple profiles, for example suppose you want test and production configs.

Warning

values specified in ini files will always override all other methods unless you use skip_ini when initialising a session.

[test]
TOKEN = 632gd7yb9squd0q8sdhq0s8diqsd0nqsdq9sdj
NODES = 20

[production]
TOKEN = 632gd7yb9squd0q8sdhq0s8diqsd0nqsdq9sdj
NODES = 200000

These could then be used later:

from pyqalx import QalxSession
qalx = QalxSession(profile_name="test")
qalx.item.add({"node_count":qalx.config['NODES']})

When testing bots or processing functions and then once you are prepared to submit it to production simply change to

>>> qalx = QalxSession(profile_name="production")

Environment variables

When a configuration is loaded it will look at all the system and user environment variables and will extract any that start with the name QALX_USER_ or QALX_BOT_ and save them in the config with the leading identifier stripped. For example, with a bot like this:

from pyqalx import Bot
bot = Bot("my bot")

@bot.process
def use_local_file(job):
    local_file = open(job.session.config['LOCAL_FILE_PATH'], 'r').read()
    remote_file = job.entity.get_item_data("REMOTE_FILE").read()

You could use the same code on multiple machines and just change the value of the QALX_BOT_LOCAL_FILE_PATH on each machine to set the correct location.

Subclassing Config

This is the most pythonic way to introduce additional configurations and override the defaults. Any session can be started with a custom config class:

from pyqalx import QalxSession
from pyqalx.config import UserConfig

my_config = {
    "N_THINGS": 50,
    "MSG_BLACKOUTSECONDS": 400
            }

class MyUserConfig(UserConfig):

    @property
    def defaults(self):
        config = super(MyUserConfig, self).defaults
        config.update(my_config)
        return config

qalx = QalxSession(config_class=MyUserConfig)
thing_item = qalx.item.add({"how_many":10*qalx.config['N_THINGS']})