Saturday, December 3, 2022

Downloading Files from FTP and Doing Something With Them In D365FO

 Although D365FO is now thoroughly modern and web based, with all the good things that entails, we still have to interface with some less modern systems. Sometimes those systems are literally older than we are. There are some middleware solutions that can help bridge the gap but we have quite a few tools in D365FO that can help us. Let's talk about FTP for a bit. The tried and true battle horse of Odin himself, this is never going to go away, ever. So, let's see what we can do from inside D365FO with no C# in play; just some standard tools available from x++. Consider the very simple example:

using System.Net;

class AAX_FTPDownloadSimple
{

    public static void main(Args args)
    {
        System.Net.WebClient webClient = new System.Net.WebClient();
        webClient.Credentials = new NetworkCredential("userName", "password");
        webClient.DownloadFile("ftp://nadummy.sharefileftp.com/test/test.txt", @"C:\temp\test.txt");
        info("complete");
    }

}

3 line of code to download a file and place it in an arbitrary location. It's not particularly helpful as we have to know the source of the file and we have lots of hard coding. However, we now have a file that we could use TextIo on, if we wanted. I'm positive this is a dev box only example as you wouldn't have access in that way to the underlying server disk on anything that isn't a one box. Let's look at something a little more feature rich.

using System.IO;
using WebClient = System.Net.WebClient;
using NetworkCredential = System.Net.NetworkCredential;

class AAX_FileDownloadIntermediate
{
    public static void main(Args args)
    {
        TextIo textIo;

        str inputfilepath = System.IO.Path::GetTempPath() + System.IO.Path::GetRandomFileName();
        str ftphost = "nadummy.sharefileftp.com";
        str ftpfilepath = "/test/test.txt";

        str ftpfullpath = "ftp://" + ftphost + ftpfilepath;

        using (WebClient request = new WebClient())
        {
            request.Credentials = new NetworkCredential("userName", "password");

            System.Byte[] fileData = request.DownloadData(ftpfullpath);

            using (System.IO.FileStream file = System.IO.File::Create(inputfilepath))
            {
                file.Write(fileData, 0, fileData.Length);
                file.Close();
            }
            info("download Complete");
        }

        textIo = new TextIo(inputfilepath, "r");
        
        info(con2Str(textIo.read()));

        TextIo.finalize();

        System.IO.File::Delete(inputfilepath);

        info("Complete");
    }

}

Similar to that last one but we're handling data in a file, rather than a file plus a few other tweaks. First, we're using webClient.DownloadData rather than webClient.DownloadFile then writing that to a FileStream. Next, we're taking that FileStream and writing it to temporary (.NET) disk storage. Additionally, we're opening the file for the FileStream using TextIo, which we've had for years, so we've working with something familiar. Once done, we are deleting the temp file. I'm not sure if this would work in a non-onebox environment but .NET may be aware of its deployment type and sort itself out. I can't really say so definitely test in a Tier 2 environment. Finally, we're using using in a way most x++ people may be unfamiliar with. Next, let's keep on adding stuff:

using FTPWebRequest = System.Net.FtpWebRequest;
using WebRequest = System.Net.WebRequest;
using NetworkCredential = System.Net.NetworkCredential;
using WebRequestMethods = System.Net.WebRequestMethods;
using Stream = System.IO.Stream;

class AAX_FileDownloadAdvanced
{
    public static void main(Args args)
    {
        TextStreamIo textStreamIo;

        System.Byte[] buffer = new System.Byte[10240]();
 
        System.IO.Stream fileStream = new System.IO.MemoryStream();

        str targetFile = System.IO.Path::GetRandomFileName();
        FtpWebRequest request = WebRequest::Create("ftp://nadummy.sharefileftp.com/test/test.txt");
        request.Credentials = new NetworkCredential("userName", "password");    

        using (Stream ftpStream = request.GetResponse().GetResponseStream())
        {
            int read = 0;
            
            while(true)
            {
                read = ftpStream.Read(buffer, 0, buffer.Length);
                if(read > 0)
                {
                    fileStream.Write(buffer, 0, read);
                    info(strFmt("Downloaded %1 bytes", fileStream.Position));
                }
                else
                {
                    break;
                }
            }
        }

        textStreamIo = textStreamIo::constructForRead(fileStream);
        
        info(con2Str(textStreamIo.read()));
      
        textStreamIo.finalize();
        fileStream.Dispose();

        info("Complete");
    }

}

Here we have quite a few differences. First, we're using a WebRequest rather than a WebClientWebRequests support encryption so stuff like SFTP or FTPS becomes an option. Next we're grabbing chunks from our source file so we can report on it's progress. If we had a large file, this would be helpful to log what is happening. Next, we're taking the data we placed in a Stream and are interacting with it without placing it in any type of local storage, temporary or otherwise. You could use File::SendFileToTempStore() and File::UseFileFromURL() to store/retrieve in/from blob storage if you like. Finally, we are passing the stream to a stream implementation of TextIo as TextStreamIo.

D365 – Data Entity Method Call Sequence

 




1. initValue
2. validateField
3. validateWrite
4. update
4.1. doUpdate
4.1.1. persistEntity
4.1.1.1. doPersistEntity
4.1.1.1.1. initializeDataSources
4.1.1.1.1.1. initializeEntityDataSource
Note: initializeDataSource is called once for each DataSource in Entity.
4.1.1.1.2. mapEntityToDataSources
Note: initializeDataSource is called once for each DataSource in Entity.
4.1.1.1.3. saveDataSources
4.1.1.1.3.1. updateEntityDataSource
4.1.1.1.4. mapEntityToDataSource (maybe for another record)
4.1.1.1.5. saveDataSources
4.1.1.1.5.1. updateEntityDataSource for update operation and (insertEntityDataSource for insert)
4.1.1.1.5.1.1. mapDataSourceToEntity
4.1.1.1.5.1.2. doSaveDataSource
4.1.1.1.5.1.2.1. updateDataSource
4.1.1.1.5.1.2.1.1. preupInsertDataSource
4.1.1.1.5.1.2.1.1.1. validateWrite of table
Plus:
postLoad
This method is called during the export for setting the value to unmapped fields after entity is downloaded to datasource.

EXPORT:
       Entity- postLoad()
       staging - insert()
       Entity- postLoad() - depends on records

IMPORT:
       staging - postLoad()
       Entity - postLoad()
       Entity - initValue()
       Entity - validateField() - depends on no of fields
       Entity - validateWrite()
       Entity - insert() / update()
       Entity - persistEntity()
       Entity - initializeEntityDataSource()
       Entity - mapEntityToDataSource()
       Entity - insertEntityDataSource() / updateEntityDataSource()
       Entity - mapDataSourceToEntity()
       staging - postLoad()

Here are some method’s calls during Import:

defaultCTQuery
copyCustomStagingToTarget
postGetStagingData
preTargetProcessSetBased
postTargetProcess

My notes:

  1. The postLoad method is called also by import! Since postLoad is the recommended place to set values for the virtual fields, this potentially slow-down the import process unnecessarily.
  2. Be careful by using the postTargetProcess method! This method is called at the end of EACH thread/task-bundle if you are using the “Import threshold record count” option.
  3. Previously, you could add postTargetProcess only to a newly created entity, but now you can extend any entity using CoC

Extension is simple.

[ExtensionOf(tableStr(DataEntity))]
final public class DataEntity_Extension
{

    public static void postTargetProcess(DMFDefinitionGroupExecution _dmfDefinitionGroupExecution)
    {
        // Do no call next
        //logic here
    }
}

Please note that this can be done only MDM scenarios but not via ODATA because ODATA updates, inserts records row by row and there is no post event\method to use. You might wish to use OData action to perform post actions on records.