The.log batches can be transmitted in 2 ways.

Direct File Upload to Google Cloud Storage

  • To upload.log batch files to GCS, you must first apply for BigQuery permissions.

  • GCS Bucket List

    Operating Environment

    Bucket Address

    Description
    Live Service gs://snapshot_upload_live Bucket for uploading log batch data for service operating environment
    Sandbox gs://snapshot_upload_sandbox Bucket for uploading log batch data for sandbox environment

  • (Mandatory) File Naming Convention

    • CompanyName_CategoryValue/YYYY/MM/dd/CompanyName_CategoryValue_YYYY_MM_dd_HH_mm_ss_uuidRandom.log
      • It is mandatory to follow the file naming convention.
      • The date and uuid in the file name are for checking the storage date and preventing duplication.
    • The file extension is “.log”.
    • Example)
      • com2us_nbasuperstars_team/2023/03/24/com2us_nbasuperstars_team_2023-03-24_05_55_00_c4c9c6e0-4093-4e29-853f-2d0d9608ca3b.log

  • File Format

    • Files should be composed of logs for each table and data should not be mixed with different categories.

    • Newline-separated json file (sample)

    • Files should not be compressed.
    • File size should be controlled to not exceed 1G.
      • When uploading multiple files, it’s better to have fewer files with larger sizes.
      • Uploading too many files at once can cause errors (less than 10 per second).

  • GCS File Upload Guide

Log Transmission via fluentD

Log batch cannot be transferred to the server receiving via streaming FluentD.
For detailed information about using FluentD, refer to fluentd method page.

  • fluentD Transmission Precautions

    • Chunk file – limit is 10Mbyte. Files exceeding this limit are automatically deleted, so take care.
    • If line-by-line transmission (less than 10Mbyte) is not possible, it is recommended to use the google cloud storage file upload.

  • fluentD Tag Rule

    • Tag creation rule
      • CompanyName_Table Name(category value)_snapshot
    • Example)
      • Use the tag created by combining the company name and category value, including _snapshot
        • com2us_logname_snapshot

References

  • Regardless of the method of transmission, log batch files are stored in GCS for a certain period.
    • Retention period: 90 days
    • Files older than 90 days are deleted and cannot be recovered. (Data in BigQuery is still queryable)
  • Files are kept for purposes such as reloading data into BigQuery later or verifying data omissions.