Free random fake name generators for testing FileGateway

I thought of a site that allows one to generate records of fake names with other general fields. You can generate 50,000 records at a time, for free. Visit here for more information.
http://www.fakenamegenerator.com/
With 50,000 records we can test the performance and collect realistic performance data via Hawk.
I have modified the Data Format configuration as well as the Mapper activity to accommodate this change.
The Data Format describing the input csv file.

<?xml version="1.0" encoding="UTF-8"?>
<xsd:element xmlns:xsd="http://www.w3.org/2001/XMLSchema" name="emp">
    <xsd:complexType>
        <xsd:sequence>
            <xsd:element name="empNum" type="xsd:int"/>
            <xsd:element name="gender" type="xsd:string"/>
            <xsd:element name="firstname" type="xsd:string"/>
            <xsd:element name="midinit" type="xsd:string"/>
            <xsd:element name="lastname" type="xsd:string"/>
            <xsd:element name="stradd" type="xsd:string"/>
            <xsd:element name="city" type="xsd:string"/>
            <xsd:element name="state" type="xsd:string"/>
            <xsd:element name="zip" type="xsd:int"/>
            <xsd:element name="cntry" type="xsd:string"/>
            <xsd:element name="email" type="xsd:string"/>
            <xsd:element name="phone" type="xsd:string"/>
            <xsd:element name="mommaiden" type="xsd:string"/>
            <xsd:element name="dob" type="xsd:string"/>
            <xsd:element name="cctype" type="xsd:string"/>
            <xsd:element name="ccnum" type="xsd:long"/>
            <xsd:element name="cvv2" type="xsd:string"/>
            <xsd:element name="ccexp" type="xsd:string"/>
            <xsd:element name="nationalid" type="xsd:string"/>
            <xsd:element name="ups" type="xsd:string"/>
            <xsd:element name="job" type="xsd:string" minOccurs="0"/>
            <xsd:element name="domain" type="xsd:string"/>
        </xsd:sequence>
    </xsd:complexType>
</xsd:element>

Here is the updated BusinessWorks project file. Just to be safe, I would let you obtain your own copies of fake names rather than distributing something that I am not sure I should.
Well, nothing is better than putting your hands into the tasks, try’em out!
Cheers, happy SOA'ing!
Note1 : When you generate your own list of fake names, please ensure that you select all fields except the password field. They have been mapped in the process definition.
Note 2:You also need to modify the designer.tra file to allow a large heap size. Locate the "tibco.env.HEAP_SIZE" variable in the designer.tra and assign it a value of 1024M

tibco.env.HEAP_SIZE 1024M

A TIBCO BusinessWorks-based file gateway – Part 2

In the previous article, we went through the steps to create a TIBCO BW process that parses a flat CSV file and transforms the data format from CSV to XML. Along the process we also transformed the data model, concatenating the firstname and lastname fields into a single element specified in the employee.xsd.
This article adds a file poller process with Render XML activity and sets up a few global variables for configurations. At the end of this article, we will have something like this.


For now, the following are configurable:

• Inbound directory: DirInboundFolder (default to 'c:\fgw\inbound\')
• Outbound directory: DirOutboundFolder (default to 'c:\fgw\outbound\')
• Semaphore file extension: InboundSemaphoreExt (default to 'done')
• Semaphore extension separator: InboundSemaphoreExtSeparator (default to '.')
• Search pattern: InboundSemaphorePattern (default to '*')

Steps
This article assumes that you already knew how to perform basic tasks in TIBCO Designer.
1) Add a new folder under the FileGateway root folder.
2) Add a new process definition with a name PollEmpsCSV
3) Double-click the newly created process definition. Add a File Poller activity into the process. Note that the original ‘Start’ activity is now replaced by the File Poller activity.
4) Configure the File Poller activity.

In the Configuration tab, paste the following string into the File Name field:

%%DirInboundFolder%%%%InboundSemaphorePattern%%%%InboundSemaphoreExtSeparator%%%%InboundSemaphoreExt%%

The global variable name surrounded by double % signs will be resolved during run time.


In the Advanced tab:


5) Expand the ParseAndTransform folder in the Project Panel. Drag the 'ParseAndTransform' process into the PollEmpsCSV process. Wire the PollDumpCompletion activity, ParseAndTransform process and the End activity in the following order.
 
6) An input parameter has already been defined earlier in the 'Start' activity of the ParseAndTransform process. This parameter holds the fully qualified name of the file to be parsed by the parser. While the Poller activity polls for the semaphore file, we will need to construct the filename required. This step assumes that the semaphore file without the extension is the filename to be processed.

In the input tab of ParseAndTransform sub-process, click on the pencil icon to bring up the XPath editor.



Paste the following XPath formula into the formula pane.

tib:substring-before-last($PollDumpCompletion/ns:EventSourceOuputNoContentClass/fileInfo/fullName,string-length($_globalVariables/pfx:GlobalVariables/InboundSemaphoreExtSeparator) + string-length($_globalVariables/pfx:GlobalVariables/InboundSemaphoreExt))

The above XPath formula returns the filename required by the ParseEmpsCSV activity inside the sub-process. If the semaphore filename is
c:\fgw\inbound\emp001.txt.done
then the input filename to the parser activity will be
c:\fgw\inbound\emp001.txt
There are different ways to implement semaphore. Another alternative is to rename the source file when the transfer/file creation by the legacy system has completed. Semaphore is required when writing large file that takes time to complete. This is to avoid the process from reading an incomplete file.

7) The ParseAndTransform sub-process provides 2 output elements in its output schema defined in its 'End' activity. Refer step 13 of part 1 of this article.
Add a render XML activity followed by a write file activity. The write file activity will use the filename specified in the 'outputFileName' element of the ParseAndTransform output schema.

8) For the sake of simplicity, add a 'Catch' activity that catches all unhandled exceptions.


The final process looks like this.


Download the entire project here and play with it!
There are a lot of works to be done for this gateway to be useful. The following considerations are in mind.

• The semaphore file cleanup mechanism
• The separation of write file activity into a sub-process
• Error logging and resolutions
• The 'execute-only-once' restriction
• Refactoring of global variables, shared variables, and other considerations such as implementation of checkpoints for fault tolerance and load balancing.

Happy testing.