Wurker Documentation

• Colony: a cluster of Wurker nodes.
• Queen: the controller node of a Colony. May be the only node.
• Drone: a worker node of a Colony.
• Bee: a single job, either the job definition, or the running process, itself.
• Command: a single command (script or Python module) that Bees reference.
• Cron: a single scheduled or supervised definition of how and when to run.
• Hive: the set of currently-running Bees.

• Sting: inject dependencies into Wurker, including wurker.toml, bees, settings, models, seeds, and secret data, such as data/user.json.
• Seed: declare Commands, Bees, Crons and Maps to be added, updated, or deleted.
• Schedule: run short-lived Bees as scheduled on cron-type schedules.
• Supervise: run long-lived Bees as supervised and restart the process if it dies.
• Ondemand: run any enabled Bees one time immediately.
• CLI: easily add, remove, and edit Bees, Crons, and Maps from the command-line using JSON.
• UI: serve a dashboard to view and enable or disable Bees, including the Hive.
• API: perform many of the same actions as CLI or UI via API request. Used by the Queen to talk to Drones.

Wurker requires a minimum Python version of 3.10. A virtual environment is strongly recommended.

Wurker uses two databases:

1. Remote Configuration DB: by default a sqlite3 file in data/wurker-remote.db (defined in etc/wurker.toml).
2. Local DBs: sqlite3 file in data/wurker.db keeps track of Wurker processes and Wurker UI users.

Both databases are configured through SQLAlchemy, so you can switch either one to any supported DBMS by editing [local_db] or [remote_db] in wurker.toml. SQLite works out of the box. For MySQL or PostgreSQL, install the optional extras:

pip install wurker[mysql]
pip install wurker[postgres]

To use the default MySQL DBAPI, first install the Python 3 and MySQL development headers and libraries:

Wurker UI uses vanilla JavaScript, but it includes a JSON editor widget. The editor assets are vendored under ui/static/vendor/jsoneditor so that wurker-ui run works without requiring Node.js. If you prefer to update the JSON editor from npm for development, you can install it locally and refresh the vendored files.

If you are deploying to production and want to use the web app UI, you will need to install nginx.

Further instructions may be found in the Production section.

Wurker may be installed from PyPi using Pip or similar package manager. A Python virtual environment is strongly recommended:

pip install wurker

If you plan to use MySQL or PostgreSQL, install the optional extras:

pip install wurker[mysql]
pip install wurker[postgres]

Alternatively, Wurker may be cloned and run as a Hatch project.

• Install pipx
• Install hatch

Next, install system dependencies, as documented in System Dependencies.

After hatch and all system dependencies are successfully installed, cd into the wurker directory and do:

hatch env create
hatch shell

Existing version 0.0.X Wurker installs will need to meet new system requirements.

Before running the test suite, you need to initialize the test environment with the default stinger and seed data:

wurk admin init

Or, if you prefer the explicit steps:

wurk admin stinger sting
wurk admin seed

After initializing the test environment, you can run the full test suite with:

hatch run pytest src/wurker/tests/ -v

Or run individual test files:

hatch run pytest src/wurker/tests/test_00_core.py -v

You may use Stingers to inject additional Python requirements into etc/requirements.txt. You, or your CI/CD, will need to then perform additional package installation:

pip install -r etc/requirements.txt

You can check that installation worked by executing the following commands:

wurk --help

Use the default Stinger to inject basic dependencies, including test Seeds and Bees:

wurk admin init

Or run the explicit stinger command:

wurk admin stinger sting

This will inject local resources. For custom dependency-injection, see the Stingers section.

The default database set in etc/wurker.toml will be a SQLite file named wurker.db. See the Configuration section for details.

After injecting resources, you can seed the local and remote configuration databases:

wurk admin seed

Finally, and optionally, you can enable some of the test Bees:

wurk edit bee test_scheduled_bee <<EOF
{"enabled": true}
EOF

You can see that test_scheduled_bee was enabled with the command:

wurk show

And you can run the Hive in the foreground with the command, while logging to stderr:

wurk run

If you choose to log to syslog, instead, you can open a separate terminal and do:

journalctl -f

You can also override the wurker.toml log setting from the command-line. Type C-c to stop running, then run again but this time, force logging to dump to stderr:

wurk --log stderr run

Running Wurker this way is only for development and devops use. In production, Wurker will run as a systemd service, and we will use wurk in parallel to access the same runtime and configuration data as the service.

For example, when running Wurker as a service, we can follow all the logs this way:

journalctl -u wurker -f

But when running wurk run we cannot follow -u wurker because that is the name of a systemd unit which does not exist. Instead, we must follow a text name, like -t Wurker.

If all else fails, follow and filter logs like:

journalctl -f | grep Wurker

Since our Test Run example is a Queen (all singleton nodes are Queens) we can also serve the Dashboard GUI.

Open a new terminal, so you don't interrupt the running Wurker. Then in the new terminal, activate your virtual environment and run the command:

wurker-ui run

Then browse to specified URL.

The default development credentials are wurkeradmin/wurkeradmin.

To create your own Wurker UI credentials, see the Stingers section.

To configure Wurker locally, it is enough to use the default wurker.toml file:

[wurker]
name = "Wurker"
app_id = "6b018c2b-77b7-40ca-bb82-cc8378252f67" # Must be UUID
env = "local"                                   # 'local', 'development', or 'production'
log = "stderr"                                  # 'stderr' or 'syslog'
loglevel = "debug"
is_queen = true
debug = true

[ui]
domain = ""                                     # Default: localhost
workers = 3                                     # Default: 3

[local_db]
hostname = ""                                   # Default: local sqlite file
username = ""                                   # Default: local sqlite file
password = ""                                   # Default: local sqlite file
database = "wurker.db"                          # Default: local sqlite file 'wurker.db'
dbdriver = "sqlite"                             # Default: 'sqlite'
port = 0

[remote_db]
hostname = ""                                   # Default: local sqlite file
username = ""                                   # Default: local sqlite file
password = ""                                   # Default: local sqlite file
database = "wurker-remote.db"                  # Default: local sqlite file 'wurker-remote.db'
dbdriver = "sqlite"                             # Default: 'sqlite'
port = 0

While the wurker.toml file alone is sufficient for a few Bees abuzz, Wurker operators may additionally configure custom Python and JSON dependencies using Stingers.

Wurker configuration of Commands, Crons, and Bees are defined in remote database tables within the schema specified by [remote_db] in the previous Local Config section.

• A Command is either a path to a command (ex: /usr/bin/ls), or the name of a module in bees/.
• A Supervised Cron is a wurker_cron that has a null cron field.
• A Scheduled Cron is a wurker_cron with a valid cron field in crontab notation.
• An Orphaned Bee is any Bee that is not mapped to a Cron, either supervised or scheduled.
• A Disabled Bee is any Bee that is not enabled. Bees may be both orphaned and disabled.

Because Wurker permits the use of internal PyBees, it also permits internal configuration of PyBees and PyBee-compatible modules.

Wurker and the Wurker UI run as systemd services. In addition, Wurker UI requires an nginx service.

After configuring Wurker and Bees, you may also configure the systemd services using the service command. Then either or both of wurker and wurker-ui services will be running in the background.

Their logs will be available using journalctl.

See the Service section for full details.

The following are only simple examples for pre-configured services:

wurk admin service wurker
wurk admin service ui

Some subcommands are only available via the Command-Line Interface (CLI). In particular, you can only perform dependency injection, seed the database, configure services, and run Bees via the CLI.

The other subcommands are available via CLI or API. Most API subcommands are also available via the UI in a more limited fashion.

Seed files are JSON files that used to migrate data without dropping the remote database schema. Seed files can be used to add, update, or delete zero or more Commands, Crons, Bees, and/or Maps.

A Seed file is a JSON object with at least one of wurker_command, wurker_cron, wurker_bee, or wurker_bee_cron items. All of these are JSON arrays containing objects, except wurker_bee_cron, which is an object.

See Default Seed JSON for a real example of adding and scheduling Commands, Crons, and Bees.

See Seed JSON Schema for a formal declaration that can be used to validate Seed JSON data.

Stingers are a powerful tool that may include Seeds, as well as additional dependencies to be injected, like PyBee modules, custom DB models, custom Python libraries, etc.

The Seeds permit us to declare a complete configuration for all our Commands, Crons, and Bees that are injected during the sting operation.

The wurk admin stinger sting options should be familiar concepts, by now:

• ROOT: Required: the root of the Stingers you want to choose from.
• SLUG: Required: the directory of the Stinger you want to inject.
• NODE: Optional: the directory of the node (queen, drone1, drone2, etc) you want to inject.
• ENV: Optional: the directory of the environment (production, development, etc) you want to inject.

Colonies are Wurker clusters: multiple hosts (hardware or virtual machines) running in tandem.

Each separate host in a Colony runs a Wurker instance, identified by the app_id in the wurker.toml file. All nodes in a Colony share the same database schema. Only one of these instances in the Colony can be the Queen. The others are Drones.

Stingers make it much easier to declare Queens and Drones, and when used in conjunction with version control and a CI/CD system, they provide a highly reproducible solution to scaling Wurker on conventional, low-cost cloud infrastructure.

Consider the final example from the previous section on Stingers, and note the colony directory.

Like the wurker directory, the colony slug is special. It indicates that one or more NODE directories will be found inside. And each NODE directory may have a wurker directory, or one or more ENV directories. Or both wurker and ENV directories.

Every Wurker instance may serve an API, and it may also serve the Wurker Dashboard UI, if it's a Queen.

During the startup phase, all versions were considered BUILD versions until feature completeness is achieved.

That means we set MAJOR = 0 and MINOR = 0 for all early versions.

After v1.0.0, apply the following criteria when determining which version value to increment:

• MAJOR: A new subcommand or integration is added, or major code revisions have occurred. Increment rarely.
• MINOR: Any release candidate that has been deemed "production-ready". Increment often.
• BUILD: For development use only. It should be excluded in releases, except released patches.

The road map items should be moved to the Change Log as they are completed.

Added:

• Added wurk admin init to run default stinger injection, seed data, and optional status in one step
• Added --username/--password to wurk admin init to update data/user.json before seeding

Changed:

• Default remote SQLite database is now data/wurker-remote.db (local remains data/wurker.db)
• Settings bootstrap is now lazy to avoid import-time filesystem side effects
• UI auto-initializes its SQLite schema on first run without wiping existing data
• Standardized runtime timestamps to timezone-aware UTC
• Documentation updated for new init flow, split DB defaults, and vendored UI assets

Fixed:

• Alembic migrations now target the configured remote DB URL for SQLite
• Bee run now tolerates missing args; PyBee avoids mutable default args
• Hive logger no longer raises on exception logging
• Packaging no longer references a missing src/wurker-ui module
• Replaced croniter with croniter-rs for Python 3.12 compatibility

Changed:

• Dropped docs/README.md and dropped CHANGELOG.md TOC
• Deleted root README.md, relying on org mode rendering
• Updated documentation and changelog structure

Changed:

• Dropped and ignored Gemfile.lock

Changed:

• Pulled v0.1.16
• Updated docs, added CHANGELOG, and fixed a couple small testing bugs
• Updated READMEs

Added:

• More flexible control/ondemand check window

Changed:

• Debugging control & ondemand out-of-loop
• Debugging PyBee, dispatch and models
• Moved load_plugins to only non-admin commands
• Fixed regression bugs involving PyBee.run()

Added:

• CommandNotReady exception handling for some commands
• User/group support for systemd service templates and logic
• Working directory configuration for systemd services

Changed:

• Fixed path resolution for scripts directory
• Removed extra output from admin service
• Updated documentation for test.pypi.org release

Changed:

• Major refactor to use Hatch project manager for PyPI packaging
• Restructured project layout with all modules under src/wurker
• Converted from argparse to Click for CLI framework with nested subcommands
• Implemented plugin system for injected dependencies
• Updated SQLAlchemy to v2.0 syntax
• Revised CLI commands to use --slug option and --drone filtering
• Enhanced service configuration with user/group support
• Improved exception handling for CommandNotReady errors
• Updated documentation and configuration for system dependencies

Added:

• CommandNotReady exception handling for some commands
• User/group support for systemd service templates and logic
• Working directory configuration for systemd services

Changed:

• Fixed path resolution for scripts directory
• Removed extra output from admin service
• Updated documentation for test.pypi.org release

Added:

• Logic to restrict some CLI commands until resources are marshalled
• Bundled node_modules for UI dependencies

Changed:

• Various bug fixes and improvements
• Removed sudo-required functionality
• Updated documentation

Added:

• No specific changes noted in commit history

Added:

• Bundled node_modules for UI dependencies
• Updated pyproject.toml for Hatch configuration
• Include src/wurker/etc/ data in package

Changed:

• Updated documentation in preparation for test.pypi.org release

Added:

• Plugin system implementation for injected dependencies
• Support for MySQL and PostgreSQL databases

Changed:

• Major revisions to multiple components for improved functionality

Added:

• No specific changes noted in commit history

Added:

• No specific changes noted in commit history

Added:

• No specific changes noted in commit history

Added:

• Complete transition to Hatch project structure
• Full Click-based CLI implementation
• Enhanced documentation system
• GitLab CI/CD integration

Changed:

• Refactored to use modern Python packaging standards
• Updated all dependencies and configurations for PyPI distribution

Added:

• Add overrun column to wurker_cron table (model).
• Refactor UIs, seeder and seeds. Make optional everywhere.
• Add runcount field to wurker_bee_report.status_info for display.
• Wurkers should know if they're Queens or Drones. Both should have an API, but only Queens have a GUI.
• The Queen's Court should provide the same intuitive display and control of Drone data.
• Wurker UI should have a login screen and session.
• Wurker UI and API should support HTTPS.
• After login, the Queen may use Basic HTTP Auth to access Drone APIs.
• User logins should be configured via Stinger.
• Accessing Wurker UI or API (nginx) should be via HTTPS, not HTTP. Should support self-signed certs by default.
• Add wurk report subcommand with filtering.
• Add Report view to Wurker UI (separate page) with same filters as CLI.
• Add Action buttons per Bee row, and Cron table, for pre-filtered Report views.
• Implement Alembic migrations in place of current create schema logic.

Changed:

• Derive timeout instead of setting it. If the Bee has a scheduled Cron, then use that as the timeout. If it is a supervised Cron, then it should have no timeout. Ondemand should use the same timeout logic.
• Derive runcount from Cron to BaseBee.runcount for runtime reference.
• Upon timeout in _run_scheduled():
  • If bee.overrun == 0 then timeout as normal. Set report status to TIMEOUT.
  • If bee.overrun > 0 and bee.runcount < bee.overrun then increment bee.runcount, and let this Cron overrun the next Cron. Set report status to OVERRUN.
  • If bee.overrun > 0 and bee.runcount >= bee.overrun then TIMEOUT the overrun.
• Deleting a Bee or Cron should cascade into Report table.
• Add pid and timestamp Hive columns, and set them when the Bee is added to the Hive.
• Add nullable duration Hive column, which is set only when the Cron ends.
• Change wurker_bee_report.status enum to: PROCESSING, OVERRUN, COMPLETED, TIMEOUT, ERROR
• Insert record on bee.run() as PROCESSING
• Update status to OVERRUN upon allowed overrun, or to TIMEOUT if not allowed.
• Record stdout errors from Bees, and PyBee exceptions into status_info JSON.
• Every Cron ending should cause a final write, setting duration, status, and status_info.

*/ Changelog Entries