In this article you are going to learn how to use the free App FileZilla to insert data through FTP/SFTP in the EM Platform.
This might be useful for you to insert historical data using .csv files in a quick and easy way.
Remember the list of FTP/SFTP available formats.
Content of the article:
Requirements
Regarding platform Settings
To be able to insert data using FileZilla, you need to have an FTP/SFTP gateway created in the EM Platform beforehand.
To do so, follow these steps and select the FTP/SFTP gateway.
From the FTP/SFTP gateway that you create, you will need, for the purpose of the article:
- The FTP/SFTP address
- The FTP/SFTP user
- And the FTP/SFTP password
You can find the 3 variables in the settings of the gateway:
Regarding Data
- Ensure that the files are sorted with the oldest data in the first rows and the newest in the last rows.
What is FileZilla and how to get it?
FileZilla is a free and easy-to-use app that allows you to drag and drop files from your PC to an FTP/SFTP folder, like the EMS gateway's one.
You can download it here.
Setup and sending of data
Once you have downloaded the app, you will see this screen:
On the top, you will have to write the credentials of your FTP/SFTP gateway to get in.
In the middle, you will see on the left the folders and files of your PC and on the right, the folders and files inside your FTP/SFTP folder.
Now, the first thing you have to do is set up the app:
1. Write the FTP/SFTP address from your gateway as the Host.
2. Write the FTP/SFTP username as the Username in FileZilla.
3. And finally, write the FTP/SFTP password as the password in FileZilla.
4. Click on Quickconnect.
At this point, get familiar with the FileZilla Interface before following the next steps.
5. This step is different depending on which type of data you want to insert:
- If it is real-time (so current) data, stay in the root
- If it is historical data, get inside the historic folder (*)
6. Now, drag and drop the files that you would like to insert into the Platform from the left menu, or from any folder of your PC, to the right section (inside the FTP/SFTP).
7. Now let's wait until the data is processed, and you will be able to visualise it in the Platform.
(*) Order in which the files are inserted: In the root folder, only new data can be sent. If you would like to insert data from a period prior to the data already inserted, you should do so in the historic folder. For example, if it is not possible to ensure that the files are inserted in chronological order, send the data to the Historic folder directly.
FileZilla Interface
Once you have configured FileZilla, you will get inside the FTP/SFTP server with your credentials, so you will get inside the FTP/SFTP folder of your gateway. FileZilla will show you this:
Depending on the connection type, either FTP or SFTP, you will need to fill the fields like this:
FTP server connection
Host: ftp.source.dexma.com
User and password: depending on the gateway you want to access.
Port: no need to fill it.
SFTP server connection
Host: sftp://sftp.source.dexma.com
User and password: depending on the gateway you want to access.
Port: you need to indicate the port 23.
In the section of your FTP/SFTP gateway, at the right, you will see that you are on the root of the FTP/SFTP. Here is where all the real-time files end up in order to be processed.
In the root, you will visualise 3 (or 4) different folders:
Historic
The aim of this folder is to insert historical data, as it is data that can't be inserted at the root of the FTP/SFTP gateway.
When the data inside this folder is processed, you will visualise it in the processed folder.
Processed
Here is where you will be able to visualise all the files (historic and real-time) that have been processed in the last 7 days.
After 7 days, the files are compressed and stored in the archives folder.
Note that files can take at least 15min to be processed.
Not processed
You will see this folder only if you have inserted a file that has not been properly processed by the FTP/SFTP system.
Why can this happen?
It can happen for many reasons, depending on the type of parser selected in the Platform. These are some of the most typical ones:
- Error in the file name
- Error in the file format (wrong parser)
- Error in the file type (.csv, .zip...)
You can check the gateway logs to get more information on the errors.
Archives
Here is where all the processed files from each week are compressed and stored for at least 1 month so that you can recover them in case you need it.