Overview

Send data secure with sftp and spring batch

1 Comment

Sending data with Spring Batch can be done in several ways. The most used case with Spring Batch is to send the data as XML files, but this framework offers you a lot of other ways too.

One of these ways is to send the data to a secure FTP server. For this use case the Spring Integration framework has the correct interface.

Firstly you need, like in all cases to create files with Spring Batch, a marshaller with a “Reader” and a “Writer”.

The Reader:

<bean id="sftpFileReader">
  <property name="dataSource" ref="dataSource" />
  <property name="sql" value="SELECT * FROM table" />
  <property name="rowMapper">
    <bean class="de.package.rowmapper.SftpFileRowMapper" />
  </property>
</bean>

The RowMapper which is used by the Reader:

package de.package.rowmapper;
 
import java.sql.ResultSet;
import java.sql.SQLException;
 
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.jdbc.core.RowMapper;
 
import de.package.domainObjects.SftpFileObject;
 
public class SftpFileRowMapper implements RowMapper {
 
    public Object mapRow(ResultSet rs, int rowNum) throws SQLException {
        SftpFileObject fileLine = new SftpFileObject();
        try {
            fileLine.setDbField1(rs.getString("dbField1"));
            fileLine.setDbField2(rs.getString("dbField2"));
            fileLine.setDbField3(rs.getString("dbField3"));
            fileLine.setDbField4(rs.getString("dbField4"));
            fileLine.setDbField5(rs.getString("dbField5"));
        } catch (SQLException e) {
            System.out.println("Can't create data row for export File.");
        }
        return fileLine;
    }
}

The DomainObject which is used by the Reader:

package de.package.domainObjects;
 
public class SftpFileObject implements java.io.Serializable {
 
    private static final long serialVersionUID = 1L;
 
    public String dbField1;
    public String dbField2;
    public String dbField3;
    public String dbField4;
    public String dbField5;
 
    public String getDbField1() {
        return dbField1;
    }
 
    public void setDbField1(String dbField1) {
        this.dbField1= dbField1;
    }
 
    public String getDbField2() {
        return dbField2;
    }
 
    public void setDbField2(String dbField2) {
        this.dbField2= dbField2;
    }
 
    public String getDbField3() {
        return dbField3;
    }
 
    public void setDbField3(String dbField3) {
        this.dbField3= dbField3;
    }
 
    public String getDbField4() {
        return dbField4;
    }
 
    public void setDbField4(String dbField4) {
        this.dbField4= dbField4;
    }
 
    public String getDbField5() {
        return dbField5;
    }
 
    public void setDbField5(String dbField5) {
        this.dbField5= dbField5;
    }
 
}

The Writer:

<bean id="sftpFileWriter" scope="step">
  <property name="resource" value="file:path/to/file/filename.csv" />
  <property name="encoding" value="ISO-8859-1" />
  <property name="headerCallback">
    <bean class="de.package.helper.HeaderCallback" />
  </property>
  <property name="lineAggregator">
    <bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
      <property name="delimiter" value=";" />
      <property name="fieldExtractor">
        <bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
          <property name="names" value="dbField1, dbField2, dbField3, dbField4, dbField5" />
        </bean>
      </property>
    </bean>
  </property>
</bean>

The HeaderCallback is only needed, if you want to give the file, in our case a csv file, a headline, which is not created by the Reader. Then you need a small java class which creates the line for the header.

package de.package.helper;
 
import java.io.IOException;
import java.io.Writer;
import org.springframework.batch.item.file.FlatFileHeaderCallback;
 
public class HeaderCallback implements FlatFileHeaderCallback {
 
    @Override
    public void writeHeader(Writer writer) throws IOException {
        writer.write("FieldHeadline1;FieldHeadline2;FieldHeadline3;FieldHeadline4;FieldHeadline5");
    }
}

For sending the files to a SFTP server you need a SftpSessionFactory, that includes the access data of the receiving server.

<bean id="sftpSessionFactory" class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
  <property name="host" value="host.of.receiver"/>
  <property name="user" value="username"/>
  <property name="password" value="secureSftpPassword"/>
  <property name="port" value="22"/>
</bean>

Then you need a channel. This channel is used to send the data.

<int:channel id="outputChannel" />

For knowing the kind of sending, you have to tell Spring Batch the way to do. This is done by an “outbound-channel-adapter”, which includes the kind of sending way, links the channel to the SftpSessionFactory and tells the framework how the file should be named on the target server (remote-filename-generator).

<int-sftp:outbound-channel-adapter id="sftpOutboundAdapter"
  session-factory="sftpSessionFactory"
  channel="outputChannel"
  charset="UTF-8"
  remote-directory="/target"
  remote-filename-generator="fileNameGenerator" />

To make it easy, the target server should store the file with the same name like it is stored on the sending server. To do this, you need only the DefaultFilenameGenerator of Spring Integration.

<bean id="fileNameGenerator" class="org.springframework.integration.file.DefaultFileNameGenerator" />

To really send the file now, you need at least a Tasklet and a BatchJob.
For the Tasklet you need a small java class where you give the filename and the channel as parameters.

<bean id="sftpJobTasklet" class="package de.package.tasklets.SftpTasklet">
  <property name="fileName" value=" path/to/file/filename.csv" />
  <property name="sftpChannel" ref="outputChannel" />
</bean>

The javacode for sending is also really simple:

package de.package.tasklets;
 
import java.io.File;
import org.springframework.batch.core.StepContribution;
import org.springframework.batch.core.scope.context.ChunkContext;
import org.springframework.batch.core.step.tasklet.Tasklet;
import org.springframework.batch.repeat.RepeatStatus;
import org.springframework.integration.Message;
import org.springframework.integration.MessageChannel;
import org.springframework.integration.support.MessageBuilder;
 
public class SftpTasklet implements Tasklet {
 
    private String fileName;
    private MessageChannel sftpChannel;
 
    @Override
    public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
 
        File file = new File(fileName);
 
        if (file.exists()) {
            Message&lt;File&gt; message = MessageBuilder.withPayload(file).build();
            try {
                sftpChannel.send(message);
            } catch (Exception e) {
                System.out.println("Could not send file per SFTP: " + e);
            }
        } else {
            System.out.println("File does not exist.");
        }
 
    return RepeatStatus.FINISHED;
  }
 
  public String getFileName() {
    return fileName;
  }
 
  public void setFileName(String fileName) {
    this.fileName = fileName;
  }
 
  public MessageChannel getSftpChannel() {
    return sftpChannel;
  }
 
  public void setSftpChannel(MessageChannel sftpChannel) {
    this.sftpChannel = sftpChannel;
  }
 
}

At least the xml code for the BatchJob:

<batch:job id="sftpJob" restartable="false">
  <batch:step id="sftpFileGenerateStep" next="sftpFileSendStep">
    <batch:tasklet>
      <batch:chunk reader="sftpFileCreator" writer="sftpFileWriter" commit-interval="100" />
      <batch:listeners>
        <batch:listener ref="fileNameListener" />
      </batch:listeners>
    </batch:tasklet>
  </batch:step>
  <batch:step id="sftpFileSendStep">
    <batch:tasklet ref="sftpJobTasklet" />
  </batch:step>
</batch:job>

This BatchJob can be started by a CronJob on the server.

At last part I will shortly post the header of the xml file called “applicationContext.xml”, which includes all the described beans:

<xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:batch="http://www.springframework.org/schema/batch"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:int-file="http://www.springframework.org/schema/integration/file"
xmlns:int-sftp="http://www.springframework.org/schema/integration/sftp"
xmlns:int-stream="http://www.springframework.org/schema/integration/stream"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.1.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/file http://www.springframework.org/schema/integration/file/spring-integration-file-2.0.xsd
http://www.springframework.org/schema/integration/sftp http://www.springframework.org/schema/integration/sftp/spring-integration-sftp-2.0.xsd
http://www.springframework.org/schema/integration/stream http://www.springframework.org/schema/integration/stream/spring-integration-stream-2.0.xsd">

Kommentare

Comment

Your email address will not be published. Required fields are marked *