Resource¶

Resources are the heart and soul of Concourse.They represent all external inputs to and outputs of jobs in the pipeline.

Each resource represents a versioned artifact with an external source of truth. Configuring the same resource in any pipeline on any Concourse cluster will behave the exact same way. Concourse will continuously check each configured resource to discover new versions. These versions then flow through the pipeline via get steps configured on jobs.

Find out more about resources in the Concourse resource documentation.

To learn more about how Concourse resource types are actually implemented under the hood, check out implementing resource types in Concourse.

Concourse Resource Base Class¶

class concoursetools.resource.ConcourseResource(version_class)[source]¶

Represents an external input or output to a pipeline.

The source defined in a Concourse pipeline is parsed into JSON by Concourse, and will be passed to the initialiser of the ConcourseResource class.

All resource logic is contained in three methods to be overloaded: fetch_new_versions(), download_version() and publish_new_version().

Parameters:

version_class (type[Version]) – The resource parses all inputs with this version class.

Example:

A resource that looks like this:

resources:
  - name: my-resource
    type: my-resource-type
    source:
      project_key: concourse
      repo: concourse
      file_path: README.md

would translate to a resource a little like this:

class MyResource(ConcourseResource):

    def __init__(self, project_key, repo, file_path, host="https://github.com/"):
        super().__init__(MyVersion)
        self.project_key = project_key
        self.repo = repo
        self.file_path = file_path
        self.host = host.rstrip("/")

If the source contains lists or mappings, then these will be passed as list and dict types respectively. The call to super().__init__ sets the version class (a subclass of Version) to be used by the resource, ensuring that output can be properly parsed.

Tip

Note that the __init__ method has a default value for host, meaning that pipeline users need not include it in their source configuration. The parameters need not be set as attributes if they can all be combined into a single class, such as an API wrapper or other construct.

property certs_dir: Path¶

The path to the Concourse worker’s certificate directory.

Warning

This folder may not always exist, depending on how the Concourse runner was configured.

See the certificate propagation documentation for more information.

abstractmethod fetch_new_versions(previous_version=None)[source]¶

Fetch new versions of the resource.

The method will be passed the previous version (an instance of the Version class) if it exists, or None if this is the first version. It should return a list of version instances in chronological order with the oldest first, “including the requested version if it’s still valid.”.

Attention

That means that if nothing has changed, you should return [previous_version]. Since get steps are cached by Concourse, this will not kick anything off.

Important

If there was no previous version, then this method should only return the latest version, and not every version from the past. This is also the case if it is impossible to determine newer versions due to something wrong with the external resource, such as a git repo which has been force pushed and can no longer be properly compared.

Parameters:: previous_version (Optional[Version]) – The most recent version of the resource. This will be set to None if the resource has never been run before.
Return type:: list[Version]
Returns:: A list of new versions.

abstractmethod download_version(version, destination_dir, build_metadata)[source]¶

Download a version and place its files within the resource directory in your pipeline.

This method is called on a get step, and the step parameters are passed as additional keyword arguments. The method should return the version (unchanged, although technically one could alter it slightly to no real effect), and a dictionary of metadata (see Step Metadata).

Note

If the desired resource version is unavailable (for example, if it was deleted), the script must exit with error.

Example:

If the resource code looks like this:

class MyResource(ConcourseResource):

    def download_version(self, version, destination_dir, build_metadata,
                         download_metadata=False, metadata_file_name="metadata.json"):
        ...
        metadata = {
            "HTTP Status": 200,
        }
        return version, metadata

then the resource user would invoke it in the pipeline like this:

- get: my-resource
  params:
    download_metadata: true

Tip

object version returned by the publish_new_version() method is passed to this method due to an implicit get step. The pipeline user has the option to set some additional parameters for this step, and so if you intend to upload something large in your put step, it might be worth including a flag in this method to skip re-downloading that data.

Parameters:

version (Version) – The version to be downloaded.
destination_dir (Path) – A path to a folder into which resource files should be placed.
build_metadata (BuildMetadata) – Metadata associated with this build.

Return type:

tuple[Version, dict[str, str]]

Returns:

The version (most likely unchanged), and a dictionary of metadata.

abstractmethod publish_new_version(sources_dir, build_metadata)[source]¶

Update a resource by publishing a new version.

This method is called on a put step, and the step parameters are passed as additional keyword arguments. The method should return the new, and a dictionary of metadata (see Step Metadata).

Warning

The sources_dir argument does not behave the same as the destination_dir argument passed to the download_version() method. This is to more easily enable the resource to interact with other resources (and task outputs), but makes it difficult to “track down” the files relating to this resource. This is a deliberate design decision by Concourse, and you should expect users to explicitly pass the path to those files should they be needed in this method.

Example:

If the resource code looks like this:

class MyResource(ConcourseResource):

    def publish_new_version(self, sources_dir, build_metadata,
                            file_path, overwrite=False):
        ...
        metadata = {
            "HTTP Status": 200,
        }
        return version, metadata

then the resource user would invoke it in the pipeline like this:

- put: my-resource
  params:
    file_path: path/to/file.txt
    overwrite: true

Parameters:

sources_dir (Path) – A path to folder containing all resources, not just this resource.
build_metadata (BuildMetadata) – Metadata associated with this build.

Return type:

tuple[Version, dict[str, str]]

Returns:

The new version, and a dictionary of metadata.

Step Metadata¶

The download_version() and publish_new_version() methods return “step metadata”, which is displayed in the Concourse web UI like so:

This metadata dictionary should contain string-only key/value pairs. This is enforced by Concourse Tools, which casts both to strings before emitting any JSON. This is not easily overloaded, as the metadata is only ever intended for visual consumption.

Warning

It is okay to leave the metadata dictionary blank, but you must return an empty dict if so.

Tip

If you want to return metadata which is not a string, then consider “stringifying” it in a readable way:

response_codes = [200, 200, 200, 500]
metadata = {
    "response_codes": ", ".join(response_codes)
}

Warning

This metadata is only shown after a step runs successfully, and is omitted if it failed.