How to Send Log Batches

The.log batches can be transmitted in 2 ways.

Direct File Upload to Google Cloud Storage

• To upload.log batch files to GCS, you must first apply for BigQuery permissions.

• GCS Bucket List

  Operating Environment	Bucket Address	Description
  Live Service	gs://snapshot_upload_live	Bucket for uploading log batch data for service operating environment
  Sandbox	gs://snapshot_upload_sandbox	Bucket for uploading log batch data for sandbox environment

• (Mandatory) File Naming Convention

  • CompanyName_CategoryValue/YYYY/MM/dd/CompanyName_CategoryValue_YYYY_MM_dd_HH_mm_ss_uuidRandom.log
    • It is mandatory to follow the file naming convention.
    • The date and uuid in the file name are for checking the storage date and preventing duplication.
  • The file extension is “.log”.
  • Example)
    • com2us_nbasuperstars_team/2023/03/24/com2us_nbasuperstars_team_2023-03-24_05_55_00_c4c9c6e0-4093-4e29-853f-2d0d9608ca3b.log

• File Format

  • Files should be composed of logs for each table and data should not be mixed with different categories.

  • Newline-separated json file (sample)

    {"dateTime":"2021-10-11 11:27:15", "category":"game_name_login_log", "timezone":"UTC+09:00", "channel":"HIVE", "user_id":10053833008, "app_id":"com.com2us.hivesdk.android.google.global.normal", "did":5102491016, "level":0, "client_ip":"172.XX.XXX.XX", "device_name":"TINNO U319AA", "company":"C2S", "server_id":"GLOBAL-2", "server_uid":256489411, "last_login_date":"2021-10-10 08:33:11", "os_version":"11", "is_emulator":0, "game_language":"en"} {"dateTime":"2021-10-11 11:27:15", "category":"game_name_login_log", "timezone":"UTC+09:00", "channel":"HIVE", "user_id":10053833008, "app_id":"com.com2us.hivesdk.android.google.global.normal", "did":5102491016, "level":0, "client_ip":"172.XX.XXX.XX", "device_name":"TINNO U319AA", "company":"C2S", "server_id":"GLOBAL-2", "server_uid":256489411, "last_login_date":"2021-10-10 08:33:11", "os_version":"11", "is_emulator":0, "game_language":"en"} {"dateTime":"2021-10-11 11:27:15", "category":"game_name_login_log", "timezone":"UTC+09:00", "channel":"HIVE", "user_id":10053833008, "app_id":"com.com2us.hivesdk.android.google.global.normal", "did":5102491016, "level":0, "client_ip":"172.XX.XXX.XX", "device_name":"TINNO U319AA", "company":"C2S", "server_id":"GLOBAL-2", "server_uid":256489411, "last_login_date":"2021-10-10 08:33:11", "os_version":"11", "is_emulator":0, "game_language":"en"}

    1 2 3

    {"dateTime":"2021-10-11 11:27:15", "category":"game_name_login_log", "timezone":"UTC+09:00", "channel":"HIVE", "user_id":10053833008, "app_id":"com.com2us.hivesdk.android.google.global.normal", "did":5102491016, "level":0, "client_ip":"172.XX.XXX.XX", "device_name":"TINNO U319AA", "company":"C2S", "server_id":"GLOBAL-2", "server_uid":256489411, "last_login_date":"2021-10-10 08:33:11", "os_version":"11", "is_emulator":0, "game_language":"en"} {"dateTime":"2021-10-11 11:27:15", "category":"game_name_login_log", "timezone":"UTC+09:00", "channel":"HIVE", "user_id":10053833008, "app_id":"com.com2us.hivesdk.android.google.global.normal", "did":5102491016, "level":0, "client_ip":"172.XX.XXX.XX", "device_name":"TINNO U319AA", "company":"C2S", "server_id":"GLOBAL-2", "server_uid":256489411, "last_login_date":"2021-10-10 08:33:11", "os_version":"11", "is_emulator":0, "game_language":"en"} {"dateTime":"2021-10-11 11:27:15", "category":"game_name_login_log", "timezone":"UTC+09:00", "channel":"HIVE", "user_id":10053833008, "app_id":"com.com2us.hivesdk.android.google.global.normal", "did":5102491016, "level":0, "client_ip":"172.XX.XXX.XX", "device_name":"TINNO U319AA", "company":"C2S", "server_id":"GLOBAL-2", "server_uid":256489411, "last_login_date":"2021-10-10 08:33:11", "os_version":"11", "is_emulator":0, "game_language":"en"}

  • Files should not be compressed.
  • File size should be controlled to not exceed 1G.
    • When uploading multiple files, it’s better to have fewer files with larger sizes.
    • Uploading too many files at once can cause errors (less than 10 per second).

• GCS File Upload Guide

  • https://cloud.google.com/storage/docs/uploading-objects?hl=ko#storage-upload-object-python
  • Immediately after uploading the file, refreshing may make it appear as if the file has disappeared.
    • This is because the file is moved immediately after processing, and re-uploading the same file can cause data to be duplicated in BigQuery, so please take note.