Skip to content

Connector Implementation

Introduction

Though every connector built with the Jitterbit Harmony Connector SDK will be different in both concept and details, there are a few core concepts that all connectors include, covered under Connector SDK Concepts in Jitterbit Harmony Connector SDK.

This page expands on those core concepts, covering implementation details for connectors developed using the Connector SDK.

Connector

A connector should extend the BaseJitterbitConnector interface and provide a factory that Jitterbit Harmony can call to create instances of it. The base implementation includes the common methods that a connector must implement. (An alternative is to implement the JitterbitConnector interface directly.)

To indicate that the class implementing this required interface is the one to be considered as the JitterbitConnector, annotate it with an @Connector annotation and provide the class implementing its factory.

(An alternative to using an annotation is to specify the factory class in the manifest of the JAR file as the value of the Jitterbit-Connector-Factory-Class attribute. See Connector Registration for details.)

These interfaces and their methods must be implemented:

  • Connection interface and its open() and close() methods
  • ConnectionFactory interface and its createConnection(Map<String,String> properties) method

Warning

Static variables should not be used in a multi-threaded environment such as a connector. This can lead to many problems including data corruption. For example, using and accessing public static variables in a utility class can cause issues:

public class MyConnectorUtils {
  public static String accessToken;
  public static String host;
  ...
}
Instead, replace these public static variables with private instance variables and use get and set methods to access them:

public class MyConnectorUtils {
  private String accessToken;

  public String getAccessToken() {
    return accessToken;
  }

  public String setAccessToken(String accessToken) {
    this.accessToken = accessToken;
  }
  ...
}

Example Connector

A simple example of a connector:

/**
 * Example Connector.
 */
@Connector(factory = ExampleConnector.ExampleConnectorFactory.class)
public class ExampleConnector extends BaseJitterbitConnector {

  public static final ExampleConnector INSTANCE = new ExampleConnector();

  static {
    connectionFactory = ExampleConnectionFactory.INSTANCE;
  }

  @Override
  public ConnectionFactory getConnectionFactory() {
    return connectionFactory;
  }

  @Override
  public String getName() {
    return "ExampleConnector";
  }

  private static ConnectionFactory connectionFactory;

  /**
   * ExampleConnectorFactory.
   */
  public static class ExampleConnectorFactory implements
    JitterbitConnector.Factory {
      @Override
      public JitterbitConnector create() {
        return ExampleConnector.INSTANCE;
      }
  }
}

Connection Factory

Connectors are typically used to establish a connection with an endpoint. To create that connection, to facilitate its configuration, and for testing the connection, provide an implementation of a ConnectionFactory. The connection will become available to activities of the connector through the context passed to each activity.

Example Connection Factory

A simple example of a connection factory:

/**
* Factory that creates an ExampleConnection instance.
*/
public class ExampleConnectionFactory implements ConnectionFactory {

  public static final ExampleConnectionFactory INSTANCE =
    new ExampleConnectionFactory();

  private ExampleConnectionFactory () {}

  /**
   * Returns a connection to an Example endpoint,
   * created from the specified properties.
   *
   * @param props properties for configuring and
   *              creating an Example connection
   * @return the configured connection
   * @throws RuntimeException if the name or password
   *                          of the specified properties
   *                          are empty or null
   */
  @Override
  public Connection createConnection(Map<String, String> props) {
    String name = props.get("name");
    String password = props.get("password");
    String locale = !props.containsKey("locale") ?
      Locale.getDefault().toString() : "EN_US";
    if (name == null || name.length() == 0) {
      throw new RuntimeException("Name property cannot be empty. " +
        "Specify the name associated with the Example connection.");
    }
    if (password == null || password.length() == 0) {
      throw new RuntimeException("Password cannot be empty. " +
        "Specify the password associated with the Example connection.");
    }
    return new ExampleConnection(name, password, locale);
  }

  /**
   * Returns the pool size configuration.
   *
   * @return the pool size configuration
   */
  @Override
  public PoolSizeConfiguration getPoolSizeConfiguration() {
    return new PoolSizeConfiguration();
  }
}

Connection

In the previous example, the class ExampleConnection (not shown) would actually create the connection. Its requirements are determined by the particular endpoint or service being connected to, any libraries being used for that connection (such as to a database or web service), and the details of the connection.

In the Cloud Studio UI, the open() method of the class implementing the Connection interface is called when a user clicks the Test button of the connection configuration. This gives the connector an opportunity to not only create the connection but also to check that the connection works. Typically, this can be done by calling the endpoint or service and either returning a small payload or using the returned value to validate the connection.

As this method is called every time to open a connection to an endpoint if is not currently open, it’s a good idea that any test be quick and small so as to not delay any further processing.

An example of this is shown in the DropboxConnection.java of the Dropbox connector:

  /**
   * Opens a Dropbox version 2 connection.
   */
  public void open() throws ConnectionException {
    if (client != null) {
      return;
    }
    try {
      DbxRequestConfig dbxConfig = new DbxRequestConfig(appKey, locale);
      client = new DbxClientV2(dbxConfig, accessToken);
      ListFolderResult results = client.files().listFolder("");
      System.out.println("Dropbox Connection successful -> app-key: "
        + appKey + ", access-token: " + accessToken);
    } catch (Exception x) {
      x.printStackTrace();
      throw new ConnectionException(Messages.DROPBOX_CODE07,
          Messages.getMessage(Messages.DROPBOX_CODE07_MSG,
            new Object[]{x.getLocalizedMessage()}), x);
    }
  }

If there is an existing connection, the method returns immediately. Otherwise, a new client connection is created inside of a try-catch block. Once the client is created, it is tested by requesting the list of objects in the root ("") folder. The results returned are not actually checked, as a successful call is sufficient. If there is an issue with the access key that a user provides to create the connection, an exception will be triggered by the Dropbox API. This will be caught and then re-thrown with an appropriate error message for the user.

Variations of this design pattern can be used depending on the endpoint or service that the connector is working with.

Activities

The activities that a connector exposes and implements are created by classes that implement JitterbitActivity. A Jitterbit activity is a unit of work with two roles:

  • Discovery/configuration: The discovery of metadata associated with an activity and the configuration of its parameters.
  • Execution: A unit of execution that is part of a chain of operations.

Activities are declared in the manifest file as Jitterbit-Activity-* attributes, with IDs that are assigned based on the registration of the connector with Jitterbit Harmony. (See Connector Registration for details.)

Each activity class receives an @Activity annotation to register it as part of the connector, and must implement an execute() method that is passed a JitterbitActivity.ExecutionContext.

During the runtime, the operation process will invoke the activity by calling the activity’s execute(ExecutionContext) method.

The execution context contains information about the request (if present) and the payload. The activity is responsible for setting the response payload (using JitterbitActivity.ExecutionContext.getResponsePayload()), which will be handed to the next activity within the chain of operations by the process operation engine.

From the passed-in context, the activity has its connection to Jitterbit Harmony. It can obtain:

  • The parameters, if any, that the activity was configured with by the end user. For example: context.getFunctionParameters().get("folder").
  • Any connections established in the initial configuration of the connection, if the developer makes them available. For example: context.getConnection().
  • Parameters of the connector itself, if the developer makes them available.
  • The request or response payload, written to or obtained from the connection.

Configurable parameters are set by the end user in the Cloud Studio UI and are made in the configuration of the connector and its activities. They are declared in the adapter.json file included in the JAR file of the connector.

The request or response payloads of an activity are the data either written to or obtained from the connection; they are determined by the XML schema files that define those payloads, as described in the next section.

Activity Request and Response

The request and response of a connector’s activities are handled using the Java API for XML Binding (JAXB, version 2+) to generate Java classes from XML schemas. A separate XML schema file (.xsd) is used for each request or response. The mapping between the generated files and these sources is given in the sun-jaxb.episode output file.

The generated Java classes can then be imported by the classes that implement the activity’s execute() method.

For example, in the Dropbox connector’s FetchFileActivity, data is moving from Dropbox into Jitterbit Harmony. That execute() method uses a DropboxConnection and the Dropbox API to retrieve data and metadata; it then sets those values into a response object (an instance of FetchFileResponse), and then marshals the response to the response payload output stream.

Conversely, in the Dropbox connector’s PutFileActivity, data is moving from Jitterbit Harmony to Dropbox. The execute() method in that class works in the opposite direction. It unmarshals an input stream, uses the Dropbox API to upload to Dropbox, and then creates a response object (in this case, an instance of PutFileResponse) completed with values obtained from the response from Dropbox.

Each activity is also responsible for implementing the getActivityRequestResponseMetadata() method and returning an ActivityRequestResponseMetaData. The XML schema files are used to create the ActivityRequestResponseMetaData.

To assist in creating that object, a helper utility (as shown in the Dropbox connector’s DropboxUtils.setRequestResponseSchemas of DropboxUtils.java) is available to load the XML schema files and set them as the request or response.

They will then appear in the Cloud Studio UI in the data schema displayed during the final step of an activity’s configuration. If a response or request is not desired or required, it can be disregarded and no data structure tree will be built in the Cloud Studio UI for that component. The Dropbox connector Fetch File activity is an example of this; it has only a response and no request data structure.

Discovery and Metadata

As part of the lifecycle of the configuration of a connector activity, you can use a discovery process to obtain information that is required to complete the configuration.

An example of this is to obtain a table name and from this selection, obtain field names. To facilitate this, an interface in the Connector SDK (org.jitterbit.connector.sdk.Discoverable) is available that can be implemented.

When the configuration for an activity is called in the Cloud Studio UI, getObjectList() is called, allowing the connector to create and return a list of discovered objects that can be displayed in the UI. After a selection is made by the user, that selection is available in the getActivityRequestResponseMetadata() method through the activityConfigProps parameter that is passed in.

See the ProcessFileActivity of the Dropbox connector for how to use discovery in the creation of metadata.

Cloud Studio UI

The connector and its activities are configured through the Cloud Studio UI. The user interface of those configurations is specified in the adapter.json file. The actual file name of the JSON file can be changed from that default; it’s specified in the JAR file manifest.

The file specifies the UI for the connector and all of its activities. The details of the JSON file are covered in Connector SDK UI Components.

Note that the name used in the JSON file must be the same name that the connector is registered under and that is defined in the Java code. See Connector Registration for details.

Manifest

These various components (registration information for the connector and each activity, classes, third-party classpath, connector UI filename) are tied together in the MANIFEST.MF that is included in the JAR file that archives the connector. The details of the registration and the manifest are covered in Connector Registration.

Building the Connector

As usual for a Java project of this complexity, a Maven pom.xml file is essential for properly linking all imported dependencies and all the components together. When running the build, you will need to first compile the XML Schema files before the Java compiling and packaging. The appropriate Maven command is:

$ mvn jaxb2:xjc compile install

Installing

Unlike Jitterbit Harmony plugins (which are installed by uploading them to Jitterbit Harmony and allowing the platform to install them by associating the plugins with an Agent Group), Jitterbit Harmony connectors built with the SDK are installed by manually placing their JAR files in the appropriate directory of a Private Agent. If the connector is to be used on an Agent Group with more than one agent, the JAR files need to be copied to each Private Agent. If the connector depends on specific libraries that are not included in its JAR files, they need to be installed in the classpath of each Private Agent so that they can be accessed at the time that the connector is loaded by the agent.

When developing a connector, if running on Linux or macOS, we recommend using a Docker Private Agent, as it can be configured to mount as a volume local to the agent the build directory of the connector. For Windows, a Windows Private Agent can be used.

The connector directory is automatically scanned by the agent for any changes, and any modified connectors are reloaded automatically, without requiring that the agent be restarted or prompted. This speeds and simplifies the development process. Note that you should not build directly into this directory, as the intermediate build products can confuse the agent.

Connector Locations

Agent Connector Directory Path (default)
Windows Private Agent C:\Program Files (x86)\Jitterbit Agent\Connectors\
Linux Private Agent /opt/jitterbit/Connectors/
Docker Private Agent /opt/jitterbit/Connectors/
This directory is usually mapped to an external directory by the command that starts the Docker image

Complete Example

The Dropbox connector is a complete working example of these concepts. Refer to it for additional details.

If you wish to customize the Dropbox connector into your own connector using your own package and domain, you would need to update—in addition to the package names, paths, content of the Java code, and the registration of your connector—these items:

  • pom.xml: Replace usage of Jitterbit with your own domain as appropriate; update the artifact name and version
  • MANIFEST.MF: Replace usage of Jitterbit with your own name as appropriate; update the keys and IDs
  • DropboxConstants.java: Update the namespace in conjunction with the XML Schema XSD files; update the connector name
  • XML Schema XSD files: Update the target namespace, in conjunction with the DropboxConstants.java
  • adapter.json and BaseJitterbitConnector: The name field of the adapter.json and the name annotating the class extending BaseJitterbitConnector are used to name the connector. If specified in the adapter.json, the framework will use that name; otherwise, the name provided in the annotation will be used. See DropboxConstants.java and DropboxConnector.java for examples of how this happens.