Connector implementation¶

Introduction¶

Though every connector built with the Jitterbit Connector SDK will be different in both design and implementation, there are a few core concepts that all connectors include, covered under Connector SDK Concepts in Jitterbit Connector SDK.

This page expands on those core concepts, covering implementation details for connectors developed using the Connector SDK.

If you are implementing a connector with the intention of submitting it for certification by Jitterbit, source code submitted for certification must follow Jitterbit’s Java code style. See the comments on code style in the section Complete example below for the checkstyle configuration file that should be used.

Connector¶

Naming¶

When naming a connector, do not include special characters or forward slashes (/) in the name. These can cause issues when the connector is loaded in the agent.

See Adapter JSON for details on specifying the different names used in a connector.

Implementation¶

A connector should extend the BaseJitterbitConnector interface and provide a factory that Harmony can call to create instances of it. The base implementation includes the common methods that a connector must implement. (An alternative is to implement the JitterbitConnector interface directly.)

To indicate that the class implementing this required interface is the one to be considered as the JitterbitConnector, annotate it with an @Connector annotation and provide the class implementing its factory.

(An alternative to using an annotation is to specify the factory class in the manifest of the JAR file as the value of the Jitterbit-Connector-Factory-Class attribute. See Connector registration for details.)

These interfaces and their methods must be implemented:

• Connection interface and its open() and close() methods
• ConnectionFactory interface and its createConnection(Map<String,String> properties) method

public class MyConnectorUtils {
  public static String accessToken;
  public static String host;
  ...
}

public class MyConnectorUtils {
  private String accessToken;

  public String getAccessToken() {
    return accessToken;
  }

  public String setAccessToken(String accessToken) {
    this.accessToken = accessToken;
  }
  ...
}

Example connector¶

A simple example of a connector:

/**
 * Example Connector.
 */
@Connector(factory = ExampleConnector.ExampleConnectorFactory.class)
public class ExampleConnector extends BaseJitterbitConnector {

  public static final ExampleConnector INSTANCE = new ExampleConnector();

  static {
    connectionFactory = ExampleConnectionFactory.INSTANCE;
  }

  @Override
  public ConnectionFactory getConnectionFactory() {
    return connectionFactory;
  }

  @Override
  public String getName() {
    return "ExampleConnector";
  }

  private static ConnectionFactory connectionFactory;

  /**
   * ExampleConnectorFactory.
   */
  public static class ExampleConnectorFactory implements
    JitterbitConnector.Factory {
      @Override
      public JitterbitConnector create() {
        return ExampleConnector.INSTANCE;
      }
  }
}

Connection factory¶

Connectors are typically used to establish a connection with an endpoint. To create that connection, to facilitate its configuration, and for testing the connection, provide an implementation of a ConnectionFactory. The connection will become available to activities of the connector through the context passed to each activity.

Example connection factory¶

A simple example of a connection factory:

/**
* Factory that creates an ExampleConnection instance.
*/
public class ExampleConnectionFactory implements ConnectionFactory {

  public static final ExampleConnectionFactory INSTANCE =
    new ExampleConnectionFactory();

  private ExampleConnectionFactory () {}

  /**
   * Returns a connection to an Example endpoint,
   * created from the specified properties.
   *
   * @param props properties for configuring and
   *              creating an Example connection
   * @return the configured connection
   * @throws RuntimeException if the name or password
   *                          of the specified properties
   *                          are empty or null
   */
  @Override
  public Connection createConnection(Map<String, String> props) {
    String name = props.get("name");
    String password = props.get("password");
    String locale = !props.containsKey("locale") ?
      Locale.getDefault().toString() : "EN_US";
    if (name == null || name.length() == 0) {
      throw new RuntimeException("Name property cannot be empty. " +
        "Specify the name associated with the Example connection.");
    }
    if (password == null || password.length() == 0) {
      throw new RuntimeException("Password cannot be empty. " +
        "Specify the password associated with the Example connection.");
    }
    return new ExampleConnection(name, password, locale);
  }

  /**
   * Returns the pool size configuration.
   *
   * @return the pool size configuration
   */
  @Override
  public PoolSizeConfiguration getPoolSizeConfiguration() {
    return new PoolSizeConfiguration();
  }
}

Connection¶

In the previous example, the class ExampleConnection (not shown) would actually create the connection. Its requirements are determined by the particular endpoint or service being connected to, any libraries being used for that connection (such as to a database or web service), and the details of the connection.

In the Integration Studio UI, the open() method of the class implementing the Connection interface is called when a user clicks the Test button of the connection configuration. This gives the connector an opportunity to not only create the connection but also to check that the connection works. Typically, this can be done by calling the endpoint or service and either returning a small payload or using the returned value to validate the connection.

As this method is called every time to open a connection to an endpoint if is not currently open, it’s a good idea that any test be quick and small so as to not delay any further processing.

An example of this is shown in the DropboxConnection.java of the Dropbox connector:

  /**
   * Opens a Dropbox version 2 connection.
   */
  public void open() throws ConnectionException {
    if (client != null) {
      return;
    }
    try {
      DbxRequestConfig dbxConfig = new DbxRequestConfig(appKey, locale);
      client = new DbxClientV2(dbxConfig, accessToken);
      ListFolderResult results = client.files().listFolder("");
      System.out.println("Dropbox Connection successful -> app-key: "
        + appKey + ", access-token: " + accessToken);
    } catch (Exception x) {
      x.printStackTrace();
      throw new ConnectionException(Messages.DROPBOX_CODE07,
          Messages.getMessage(Messages.DROPBOX_CODE07_MSG,
            new Object[]{x.getLocalizedMessage()}), x);
    }
  }

If there is an existing connection, the method returns immediately. Otherwise, a new client connection is created inside of a try-catch block. Once the client is created, it is tested by requesting the list of objects in the root ("") folder. The results returned are not actually checked, as a successful call is sufficient. If there is an issue with the access key that a user provides to create the connection, an exception will be triggered by the Dropbox API. This will be caught and then re-thrown with an appropriate error message for the user.

Variations of this design pattern can be used depending on the endpoint or service that the connector is working with.

Activities¶

The activities that a connector exposes and implements are created by classes that implement JitterbitActivity. A Jitterbit activity is a unit of work with two roles:

• Discovery/configuration: The discovery of metadata associated with an activity and the configuration of its parameters.
• Execution: A unit of execution that is part of a chain of operations.

Though the discovery process happens first in practice, we will discuss the execution process first here as it is what determines the requirements of the discovery process.

Activities are declared in the manifest file as Jitterbit-Activity-* attributes, with IDs that are assigned based on the registration of the connector with Harmony. (See Connector registration for details.)

Each activity class receives an @Activity annotation to register it as part of the connector, and must implement an execute() method that is passed a JitterbitActivity.ExecutionContext.

During the runtime, the operation process will invoke the activity by calling the activity’s execute(ExecutionContext) method.

The execution context contains information about the request (if present) and the payload. The activity is responsible for setting the response payload (by implementing the JitterbitActivity.ExecutionContext.getResponsePayload() method), which will be then handed to the next activity within the chain of operations by the process operation engine.

From the passed-in context, the activity has its connection to Harmony. It can obtain:

• The parameters, if any, that the activity was configured with by the end user. For example, by calling the context.getFunctionParameters().get("folder") method.
• Any connections established in the initial configuration of the connection, if the developer makes them available. For example, by calling the context.getConnection() method.
• Parameters of the connector itself, if the developer makes them available.
• The request or response payload, written to or obtained from the connection.

Configurable parameters are set by the end user in the Integration Studio UI and are made in the configuration of the connector and its activities. They are declared in the adapter.json file included in the JAR file of the connector.

The request or response payloads of an activity are the data either written to or obtained from the connection; they are determined by the XML schema files that define those payloads, as described in the next section.

Activity request and response¶

The request and response of a connector’s activities are handled using the Java API for XML Binding (JAXB, version 2+) to generate Java classes from XML schemas. A separate XML schema file (.xsd) is used for each request or response. The mapping between the generated files and these sources is given in the sun-jaxb.episode output file.

The generated Java classes can then be imported by the classes that implement the activity’s execute() method.

For example, in the Dropbox connector’s FetchFileActivity, data is moving from Dropbox into Harmony. That execute() method uses a DropboxConnection and the Dropbox API to retrieve data and metadata; it then sets those values into a response object (an instance of FetchFileResponse), and then marshals the response to the response payload output stream.

Conversely, in the Dropbox connector’s PutFileActivity, data is moving from Harmony to Dropbox. The execute() method in that class works in the opposite direction. It unmarshals an input stream, uses the Dropbox API to upload to Dropbox, and then creates a response object (in this case, an instance of PutFileResponse) completed with values obtained from the response from Dropbox.

Each activity is responsible for implementing the getActivityRequestResponseMetadata() method and returning an ActivityRequestResponseMetaData. The XML schema files are used to create the ActivityRequestResponseMetaData.

To assist in creating that object, a helper utility (as shown in the Dropbox connector’s DropboxUtils.setRequestResponseSchemas of DropboxUtils.java) is available to load the XML schema files and set them as the request or response.

They will then appear in the Integration Studio UI in the data schema displayed during the final step of an activity’s configuration. If a response or request is not desired or required, it can be disregarded and no data structure tree will be built in the Integration Studio UI for that component. The Dropbox connector Fetch File activity is an example of this; it has only a response and no request data structure.

If a request is required, this can be specified in the JSON file that defines the Integration Studio UI for the connector. By declaring inputRequired for an activity in the adapter.json for the connector will force the Integration Studio UI to throw a validation error for the parent operation if there isn’t a source transformation before the usage of the activity. For example, this fragment of a JSON file shows defining the SAP activity BAPI as requiring input:

"activities": {
    "bapi": {
        "displayName": "BAPI",
        "inputRequired": true,
        "properties": [
            " . . . "
        ]
    }
}

See Connector SDK UI components for details on defining the JSON file that specifies the Cloud Studio UI.

Discovery and metadata¶

As part of the lifecycle of the configuration of a connector activity, you can use a discovery process to obtain information that is required to complete the configuration.

An example of this is to obtain a table name and from this selection, obtain field names. To facilitate this, an interface in the Connector SDK (org.jitterbit.connector.sdk.Discoverable) is available for implementation.

When the configuration for an activity is called in the Integration Studio UI, the interface’s getObjectList() method is called, allowing the connector to create and return a list of discovered objects that can be displayed in the UI. After a selection is made by the user, that selection is available in the interface’s getActivityRequestResponseMetadata() method through the activityFunctionParams parameter that is passed in.

See the ProcessFileActivity of the Dropbox connector for an example of how discovery can be used in the creation of metadata.

Further examples are included in the descriptions of the Integration Studio UI components that utilize metadata, as described in the next section.

Integration Studio UI¶

The connector and its activities are configured through the Integration Studio UI. The user interface of those configurations is specified in the adapter.json file. The actual file name of the JSON file can be changed from that default; it’s specified in the JAR file manifest.

The file specifies the UI for the connector and all of its activities. The details of the JSON file are covered in Connector SDK UI components.

Components are categorized as either basic or complex components.

• Basic components do not interact with the connector. They are used to simply receive a value from the user and return it to the connector.
• Complex components are more sophisticated and involve multiple methods and additional code in the connector to implement their discovery process and to use in execution. They are intended for solving more difficult connector UI challenges.

Note that the name used in the JSON file must be the same name that the connector is registered under and that is defined in the Java code. See Connector registration for details.

Manifest¶

These various components (registration information for the connector and each activity, classes, third-party classpath, connector UI filename) are tied together in the MANIFEST.MF that is included in the JAR file that archives the connector. The details of the registration and the manifest are covered in Connector registration.

Build the connector¶

As usual for a Java project of this complexity, a Maven pom.xml file is essential for properly linking all imported dependencies and all the components together. When running the build, you will need to first compile the XML Schema files before the Java compiling and packaging. The appropriate Maven command is:

$ mvn jaxb2:xjc compile install

Install¶

Unlike Jitterbit plugins (which are installed by uploading them to Harmony and allowing the platform to install them by associating the plugins with an agent group), Harmony connectors built with the SDK are installed by manually placing their JAR files in the appropriate directory of a private agent. If the connector is to be used on an agent group with more than one agent, the JAR files need to be copied to each private agent. If the connector depends on specific libraries that are not included in its JAR files, they need to be installed in the classpath of each private agent so that they can be accessed at the time that the connector is loaded by the agent.

When developing a connector, if running on Linux or macOS, we recommend using a Docker private agent, as it can be configured to mount as a volume local to the agent the build directory of the connector. For Windows, a Windows private agent can be used.

The connector directory is automatically scanned by the agent for any changes, and any modified connectors are reloaded automatically, without requiring that the agent be restarted or prompted. This speeds and simplifies the development process. Note that you should not build directly into this directory, as the intermediate build products can confuse the agent.

Connector locations¶

Syncing¶

Public connectors (connectors published by Jitterbit) automatically sync to a Jitterbit agent as needed. To prevent the syncing of public connectors, an environment variable (SKIP_SYNC_CONNECTORS) is available that controls syncing. Set this environment variable in the shell that is running the agent and restart the agent.

Setting SKIP_SYNC_CONNECTORS to an asterisk will stop the syncing of all public connectors:

SKIP_SYNC_CONNECTORS=*

Setting SKIP_SYNC_CONNECTORS to a comma-separated list of connectors will stop the syncing of all public connectors except for those listed:

SKIP_SYNC_CONNECTORS=Box,Magento

This last example will stop the syncing of all public connectors except for the Box and Magento connectors, which will be synced.

Complete example¶

The Dropbox connector is a complete working example of these concepts. Refer to it for additional details.

If you wish to customize the Dropbox connector into your own connector using your own package and domain, you would need to update—in addition to the package names, paths, content of the Java code, and the registration of your connector—these items:

• pom.xml: Replace usage of Jitterbit with your own domain as appropriate; update the artifact name and version
• MANIFEST.MF: Replace usage of Jitterbit with your own name as appropriate; update the keys and IDs
• DropboxConstants.java: Update the namespace in conjunction with the XML Schema XSD files; update the connector name
• XML Schema XSD files: Update the target namespace, in conjunction with the DropboxConstants.java
• adapter.json and BaseJitterbitConnector: The name field of the adapter.json and the name annotating the class extending BaseJitterbitConnector are used to name the connector. If specified in the adapter.json, the framework will use that name; otherwise, the name provided in the annotation will be used. See DropboxConstants.java and DropboxConnector.java for examples of how this happens.

Code style¶

The Dropbox connector has been formatted following Jitterbit’s Java code style. This style should be followed for any code that is submitted to Jitterbit as part of the certification of a connector.

To implement this style in your source code:

• include in your source code the Jitterbit checkstyle.xml file; and
• include in your pom.xml file a reference to the maven-checkstyle-plugin and that checkstyle.xml file. Add to the <plugins> of the <build> section of the pom.xml:

  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-checkstyle-plugin</artifactId>
    <version>2.17</version>
    <executions>
      <execution>
        <id>validate</id>
        <phase>process-test-classes</phase>
        <configuration>
          <configLocation>checkstyle.xml</configLocation>
          <suppressionsLocation>suppressions.xml</suppressionsLocation>
          <encoding>UTF-8</encoding>
          <consoleOutput>true</consoleOutput>
          <failsOnError>true</failsOnError>
          <includeTestSourceDirectory>true</includeTestSourceDirectory>
        </configuration>
        <goals>
          <goal>check</goal>
        </goals>
      </execution>
    </executions>
    <dependencies>
      <dependency>
        <groupId>com.puppycrawl.tools</groupId>
        <artifactId>checkstyle</artifactId>
        <version>6.19</version>
      </dependency>
    </dependencies>
  </plugin>
  

Refer to the Dropbox connector pom.xml for an example of using this checkstyle in a project.