EldoS | Feel safer!

Software components for data protection, secure storage and transfer

Configuring inbound transport components for use in clustered environments

Usage of inbound FTPS and SFTP transport components in clustered client-side environments (where several instances of the component do the same job in parallel) is subject for certain specifics. As a number of instances of client-side components might connect to the server simultaneously, there is a risk of picking and downloading the same files for multiple times. Considering that the components are stateless and do not maintain any lists of processed files locally (and there are just too many reasons for not doing this), the solution for this problem is not as simple as it can be imagined at first glance. Indeed, good server-side locking mechanism would do the job, still the majority of server-side implementations have restricted support for it (if support locking at all).

BizCrypto offers several tactics of prevention of duplicated downloads, mainly based on server-side file locking emulation mechanisms. While generic locking is often not available on server platforms, file renaming and/or moving can serve a magic wand and do the job of locking in most of the cases.

Tactics 1. Use the “Before Get” safe download mode.

Safe Download Mode property set to Before Get forces the component to rename files on server before downloading them. This mode can effectively be used for establishing file locking in environments that do not support them. The main idea is to adjust the name into which the files are renamed not to match the mask used by the components to look up the files.

Example 1. Consider that our goal is retrieving ZIP files from a remote location. The files may have arbitrary names, but always a .zip extension. Once every particular file is downloaded, it should be deleted from server.

Let's configure the component according to our goal:

Remote Path /home/recv/
File Mask *.zip
Delete Mode On Success
Safe Download Mode Before Get
Safe Download Name %SourceFileName%.tmp

Safe Download Name is displayed in Italics in the above table because this property can be left empty under chosen requirements. If Safe Download Name is empty, the files are assigned with <original name>.<pseudo-random GUID-like identifier> name, which doesn't match the *.zip mask. However, you might need to use this property in more complex cases.

Tactics 2. Use On Before Receive and On After Receive properties.

In some cases an advanced control over file renaming and/or other locking mechanisms is needed. For example, some FTP servers support non-standard locking commands which can be utilized by the clients, should the client be aware of their existence. On Before Receive and On After Receive properties can be used to pass commands directly to the server.

The commands accept so-called “command scripts”, which instruct the adapter which command(s) and in what order should be passed to the server. A typical command script has the following look (see how-to article for the details about composing and using command scripts).

<command type=”sftp” value=”renamefile” srcfile=”%SourceFileName%” destfile=”%SourceFileNameNoExt%.tmp” />

FTP components in addition support simplified command scripts, which can be used if a server supports custom file locking and unlocking commands:

On Before Receive = CUSTOM_LOCK %SourceFileName%
On After Receive = CUSTOM_UNLOCK %SourceFileName%

Example 2. Consider that our goal is retrieving ZIP files from a remote location. The files may have arbitrary names, but always a .zip extension. Once every particular file is downloaded, it should be moved (archived) to a "/home/processed/" directory with the original name.

Safe Download Mode alone won't do the job here, as it doesn't support file archiving after the transfer. To achieve our goal, we will extend the configuration from Example 1 with a command script assigned to the On After Receive property.

Remote Path /home/recv/
File Mask *.zip
Delete Mode Never
Safe Download Mode Before Get
Safe Download Name %SourceFileName%.tmp
On After Receive SFTP component:

<command type=”sftp” value=”renamefile” srcfile=”/home/recv/%SourceFileName%.tmp” destfile=”/home/processed/%SourceFileName%” />

FTPS component:

<command type=”ftp” value=”RNFR %SourceFileName%.tmp” acceptCodes=”350” stopOnFailure=”yes” />
<command type=”ftp” value=”RNTO /home/processed/%SourceFileName%” acceptCodes=”250” stopOnFailure=”yes” />

Please note that Safe Download Name, unlike that in the previous example, must be assigned with non-empty name now. This is because we need to know the exact names into which the files are renamed, so that we could refer to them later in command scripts.

The above two tactics are the most widely used ones due to popularity of relevant server configurations. However, depending on the specifics of your particular server environment, different renaming or blocking schemes can be chosen.

Return to the list


Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!