2024-11-27 00:51:19
When building distributed systems, microservices, or any performance-critical application, handling data efficiently is paramount. Protocol Buffers (Protobuf) by Google is a fast, efficient, and language-agnostic data serialization mechanism allowing compact and optimized binary data formats. In this article, we will dive deep into the internals of how Protobuf serialization and deserialization work in Go, explore complex data types and provide optimization tips to ensure these operations happen with minimal delay.
Protocol Buffers (Protobuf) are designed to be an efficient method for serializing structured data. By converting data into a compact binary format, Protobuf helps minimize memory consumption and bandwidth usage, making it a perfect solution for performance-critical applications such as real-time systems, distributed microservices, and mobile applications where resources are limited.
At its core, Protobuf operates based on a predefined schema, which describes the structure of the data to be serialized. This schema is compiled into specific language bindings (such as Go, Python, or Java), allowing for cross-platform communication. Protobuf’s serialization mechanism converts structured data into a highly efficient binary format, which can then be deserialized back into its original form.
Before we can serialize any data, we must define the structure of the data in a .proto file. The .proto file defines the schema, which describes how Protobuf should serialize and deserialize the data.
Here’s an example schema for a Person and Address:
syntax = "proto3";
message Address {
string street = 1;
string city = 2;
string state = 3;
int32 zip_code = 4;
}
message Person {
string name = 1;
int32 id = 2;
string email = 3;
Address address = 4;
repeated string phone_numbers = 5;
}
\ In this example:
Person contains basic fields like name, id, and email.Address is a nested message within Person.repeated keyword indicates a list of phone_numbers.Each field is assigned a unique field number, which plays a crucial role during serialization, allowing Protobuf to encode the field efficiently.
Serialization is the process of converting an in-memory Go struct into a binary format. This binary format is highly optimized for both size and speed. Let’s go over how serialization works internally and how you can optimize it for complex types in Go.
To use the schema defined in the .proto file, it needs to be compiled into Go code using the protoc compiler:
protoc --go_out=. --go_opt=paths=source_relative person.proto
\
This generates a .pb.go file, containing Go structs and methods for serialization and deserialization.
Here's an example of serializing a Person struct in Go:
package main
import (
"log"
"github.com/golang/protobuf/proto"
"path/to/your/proto/package" // Adjust the import path
)
func main() {
person := &proto_package.Person{
Name: "John Doe",
Id: 150,
Email: "[email protected]",
Address: &proto_package.Address{
Street: "123 Main St",
City: "Springfield",
State: "IL",
ZipCode: 62704,
},
PhoneNumbers: []string{"123-456-7890", "098-765-4321"},
}
data, err := proto.Marshal(person)
if err != nil {
log.Fatalf("Failed to serialize person: %v", err)
}
log.Printf("Serialized data: %x", data)
}
\ In this example:
Person message is created.proto.Marshal() is used to serialize the message into a compact binary format.\ This binary format is highly efficient, but when dealing with complex or large data, there are several ways to optimize performance.
The first step in serialization is identifying each field in the Person message, extracting its value, and determining its field number and wire type.
.proto file). For example, in the Person message, name has a field number of 1, id has a field number of 2, and so on.\ Each field is represented as a tag, which is a combination of the field number and the wire type.
A tag is encoded by combining the field number and the wire type. The formula is:
tag=(field number<<3)∣wire type\text{tag} = (\text{field number} << 3) | \text{wire type}tag=(field number<<3)∣wire type
\ For example:
name field (field number 1, wire type 2 for length-delimited) would be:tag=(1<<3)∣2=0x0A\text{tag} = (1 << 3) | 2 = 0x0Atag=(1<<3)∣2=0x0A\
This tag indicates the start of the serialized name field in the binary stream.
After determining the tag, Protobuf serializes the field’s value based on its wire type. Different wire types are encoded in different ways:
Varint encoding is used for fields with integer types (int32, int64, uint32, uint64, bool). Varints use a variable number of bytes depending on the size of the integer.
\
id field, which has a value of 150, the varint encoding works as follows:0x96 0x01 in varint format. The first byte (0x96) indicates that more bytes are part of the varint (because the MSB is set), and the second byte (0x01) completes the value.id field is serialized as:0x10 (field number 2, wire type 0 for varint)0x96 0x01 (encoded value of 150).Length-delimited encoding is used for fields that contain variable-length data, such as strings, byte arrays, and nested messages.
name field, which has a value of "John Doe", the serialization process is:"John Doe" has 8 characters.8) is encoded as a varint (0x08)."John Doe" is encoded in UTF-8 bytes: 0x4A 0x6F 0x68 0x6E 0x20 0x44 0x6F 0x65.name field is serialized as:0x0A (field number 1, wire type 2 for length-delimited)0x08 (length of the string)0x4A 0x6F 0x68 0x6E 0x20 0x44 0x6F 0x65 (UTF-8 encoded string "John Doe").Fixed-length encoding is used for fixed-width types such as fixed32, fixed64, sfixed32, and sfixed64. These fields are serialized using a fixed number of bytes (4 or 8 bytes depending on the type).
\
If the Person message had a fixed32 or fixed64 field, the corresponding value would be serialized in exactly 4 or 8 bytes, respectively, without any extra length or varint encoding.
For fields that are themselves Protobuf messages (like the Address field inside the Person message), Protobuf treats them as length-delimited fields. The nested message is serialized first, and then its length and value are encoded in the parent message.
\
For the Address field:
Address message (street, city, state, zip_code) is serialized independently.Address message.Address field is serialized in the Person message with:0x22 (field number 4, wire type 2 for length-delimited).Address message.Address message.For repeated fields like phone_numbers, Protobuf serializes each element in the list individually. Each item is serialized with the same tag but with different values.
\ For example:
phone_numbers field contains two strings: "123-456-7890" and "098-765-4321"."123-456-7890") is serialized as:0x2A (field number 5, wire type 2 for length-delimited).0x0B (length of the string).0x31 0x32 0x33 0x2D 0x34 0x35 0x36 0x2D 0x37 0x38 0x39 0x30."098-765-4321") is serialized similarly with the same tag (0x2A), length, and UTF-8 encoded string value.\ Protobuf automatically handles repeated fields by serializing each element separately with the same tag.
After all fields are serialized into binary format, Protobuf concatenates the binary representations of all fields into a single binary message. This compact binary representation is the final serialized message.
\ For example, the final serialized message might look something like this (in hexadecimal form):
0A 08 4A 6F 68 6E 20 44 6F 65 10 96 01 1A 13 6A 6F 68 6E 2E 64 6F 65 40 65 78 61 6D 70 6C 65 2E 636F6D 22 0A 0A 31 32 33 20 4D 61 69 6E 20 53 74 12 0B 53 70 72 69 6E 67 66 69 65 6C 64 12 04 49 4C 1A 09 31 32 33 2D 34 35 36 2D 37 38 39 30 2A 09 30 39 38 2D 37 36 35 2D 34 33 32 31
Protobuf provides both variable-length and fixed-length types. Variable-length encoding (int32, int64) is more space-efficient for smaller numbers but slower for large values. If you expect your values to remain large, use fixed32 or fixed64.
message Product {
string name = 1;
fixed32 quantity = 2; // Use fixed-width types for performance
fixed64 price = 3;
}
By avoiding variable-length encoding, you can speed up the serialization and deserialization process.
packed for Repeated Primitive FieldsWhen working with repeated fields, packing them can improve performance by eliminating redundant field tags during serialization. Packing groups multiple values into a single length-delimited block.
message Inventory {
repeated int32 item_ids = 1 [packed=true];
}
Packing reduces the size of the serialized message, making the serialization and deserialization processes faster.
Deeply nested structures slow down both serialization and deserialization, as Protobuf needs to recursively process each level of nesting. A flatter structure leads to faster processing.
Before (Deep Nesting):
message Department {
message Team {
message Employee {
string name = 1;
}
}
}
\ After (Flatter Structure):
message Employee {
string name = 1;
}
message Team {
repeated Employee employees = 1;
}
message Department {
repeated Team teams = 1;
}
Flattening the structure eliminates unnecessary nesting, which reduces recursive processing time.
For large datasets, it’s often inefficient to serialize everything at once. Instead, break large datasets into chunks and handle serialization and deserialization incrementally using streams.
message DataChunk {
bytes chunk = 1;
int32 sequence_number = 2;
}
service FileService {
rpc UploadFile(stream DataChunk) returns (UploadStatus);
}
Streaming allows for efficient handling of large datasets, avoiding memory overhead and delays caused by processing entire messages at once.
If you frequently serialize the same data (e.g., common configurations or settings), consider caching the serialized form. This way, you can avoid repeating the serialization process.
var cache map[string][]byte
func serializeWithCache(key string, message proto.Message) ([]byte, error) {
if cachedData, ok := cache[key]; ok {
return cachedData, nil
}
data, err := proto.Marshal(message)
if err != nil {
return nil, err
}
cache[key] = data
return data, nil
}
Caching serialized data helps reduce redundant work and speeds up both serialization and deserialization.
Deserialization is the reverse process where the binary data is converted back into a Go struct. Protobuf’s deserialization process is highly optimized, but understanding how to handle complex types and large datasets efficiently can improve overall performance.
package main
import (
"log"
"github.com/golang/protobuf/proto"
"path/to/your/proto/package"
)
func main() {
data := []byte{ /* serialized data */ }
person := &proto_package.Person{}
err := proto.Unmarshal(data, person)
if err != nil {
log.Fatalf("Failed to deserialize: %v", err)
}
log.Printf("Deserialized Name: %s", person.Name)
}
In this example, proto.Unmarshal() converts the binary data back into a Go struct. The performance of deserialization can also be optimized by applying the same techniques as serialization, such as reducing nesting and streaming large data.
When the proto.Unmarshal() function is called, several steps occur internally to convert the binary data into the corresponding Go struct.
The first thing that happens is that the binary data is read sequentially. Protobuf messages are encoded in a tag-value format, where each field is stored along with its tag (containing the field number and wire type). The deserialization process needs to parse this tag and determine how to interpret the subsequent bytes.
\
0x08 means:tag >> 3), which gives 1.0x07 (tag & 0x07), which gives the wire type (for example, 0 means varint).\ This step involves reading the tag and interpreting what type of data it represents.
Once the field number and wire type are extracted, the deserializer proceeds to read the actual field data. Each wire type dictates how the data should be interpreted.
\
Varint (Wire Type 0): This is the wire type used for most integer fields (int32, int64, bool). Varint encoding stores integers in a variable number of bytes, with smaller numbers using fewer bytes. The deserialization process reads one byte at a time, checking the most significant bit (MSB) to determine if more bytes are part of the integer.
Example:
For an id field with a value of 150, the binary representation would be 0x96 0x01. The first byte (0x96) tells Protobuf that the integer continues (since the MSB is set), and the second byte (0x01) completes the value. The deserializer combines these bytes to get 150.
Length-Delimited (Wire Type 2): This wire type is used for strings, byte arrays, and nested messages. The deserializer first reads the length of the data (encoded as a varint), and then reads that many bytes.
Example:
For the field name = "John Doe", the binary data might look like 0x0A 0x08 4A 6F 68 6E 20 44 6F 65. The deserializer first reads the tag 0x0A (field 1, length-delimited). Then it reads the length 0x08, indicating that the next 8 bytes are the string "John Doe".
Fixed-Length Types (Wire Type 1 for fixed64, Wire Type 5 for fixed32): These are used for fixed-width integers and floats, and the deserializer reads 4 bytes for fixed32 and 8 bytes for fixed64 without additional interpretation.
Once the deserializer has interpreted the field number and read the associated data, it maps the field to the corresponding struct field in Go. The deserializer performs a lookup using the field number defined in the schema to determine which Go struct field corresponds to the data it has just decoded.
\
For instance, when the deserializer reads the field with field number 1 and wire type 2 (indicating that it is a length-delimited string), it knows that this corresponds to the name field in the Person struct. It then assigns the decoded value "John Doe" to the Name field in the Go object.
\
person.Name = "John Doe"
If a field is marked as repeated, the deserializer keeps track of multiple instances of that field. For example, the phone_numbers field in the Person message is a repeated string field. The deserializer collects each occurrence of the field and appends it to the list of phone numbers in the Go struct.
\
person.PhoneNumbers = append(person.PhoneNumbers, "123-456-7890")
person.PhoneNumbers = append(person.PhoneNumbers, "098-765-4321")
When deserializing nested messages (like the Address message inside the Person message), the deserializer treats them as length-delimited fields. After reading the length, it recursively parses the nested message's binary data into the corresponding Go struct.
\
For example, in the Person message:
message Address {
string street = 1;
string city = 2;
string state = 3;
int32 zip_code = 4;
}
message Person {
string name = 1;
Address address = 4;
}
\
When deserializing the Address field (field number 4), Protobuf reads the length of the Address message, and then recursively deserializes the binary data for the Address into the Address struct inside the Person.
One of the key features of Protobuf is forward and backward compatibility. During deserialization, if the binary data contains a field that is not recognized (perhaps because it was added in a newer version of the schema), the deserializer can either store the unknown field data for later use or simply ignore it.
\ This ensures that older versions of the code can still read newer messages without crashing.
Once all fields are processed, and the binary stream is fully read, the deserialization is complete. The resulting Go struct is fully populated with the deserialized data.
\
At this point, the application can access the Person object as if it had been constructed manually in Go.
Serialization and deserialization in Protobuf are highly efficient, but working with complex types and large datasets requires careful consideration. By following the optimization techniques outlined in this article—such as using fixed-width types, packing repeated fields, flattening structures, streaming large datasets, and caching—you can minimize delays and ensure high performance in your Go applications.
\ These strategies are particularly useful in systems where efficiency and speed are critical, such as in real-time applications, distributed microservices, or high-volume data processing pipelines. Understanding and leveraging Protobuf's internal mechanics allows developers to unlock the full potential of this powerful serialization framework.
2024-11-27 00:49:17
\ The Rootstock Community is announcing the new Rootstock Hacktivator Program, allowing developers to evolve Rootstock by contributing code or creating educational content.
\ Whether you have an idea for code that would improve the Rootstock experience, or wish to produce educational guides to help developers get the most out of this sidechain, rewards are available based on the impact and value of your submission.
\ In this blog post, we explore how this program works.
\ Explore the Hacktivator Program in detail and submit your work today!
The Rootstock Hacktivator Program is designed for talented community members who want to make the ecosystem an even better place.
\ Your submissions have the potential to help the network evolve and grow, with competitive compensation on offer.
\ Contributions can be made at your own pace. All that’s required is that they are of a high quality, original, directly relevant to the ecosystem, and accurate.
Given the diverse array of dApps and use cases within Rootstock, the program welcomes contributions in many forms.
\ Perhaps you'd like to add code to the command-line interface, making it easier for Web2 and Web3 developers alike to utilize Rootstock’s decentralized infrastructure.
\ Or maybe you can build tools that enrich the experience when votes are cast using ERC-20 tokens — such as graphical representations, countdown timers, or leaderboard displays.
\ Beyond that, there are opportunities to help Rootstock support multiple programming languages including Rust, Python, and Go as the sidechain expands the software development kits — enhancing usability and accessibility for all.
Beyond coding, this program welcomes content creators who can create compelling and useful content for Rootstock newcomers and enthusiasts alike.
\ From blog posts covering the latest news and insights within the ecosystem, to in-depth written guides or tutorials for technical audiences, this is your chance to showcase Rootstock's features and use cases. Videos that walk users through tasks step by step are also encouraged.
Anyone with knowledge and expertise that can benefit the Rootstock community is encouraged to get involved.
\ Submissions will be judged based on a clear set of parameters, including:
\ From a coding standpoint, higher scores will be awarded to contributions that encourage broader adoption of Rootstock, as well as those accompanied by complementary resources and troubleshooting sections.
\ And with content, there will be greater recognition for pieces that include visual elements, high levels of clarity, excellent structure, and clear examples of real-world use cases.
Once a code submission becomes a Qualified Contribution, they will be eligible to receive rewards in one of the following tiers:
\
\ Clear guidelines have also been established on the desired length for written content and videos — reflecting the greater depth needed for advanced topics. For educational content, the rewards on offer are:
\
As you begin to work on your contribution, take time to ensure it's polished and compliant with Rootstock's evaluation criteria.
\ Once you're ready, you can submit your contribution using this Google Form.
\ The team behind the program will also ask for your name or pseudonym, and country of residence, along with your email address, wallet address, and details of the contribution you're making. A brief description of the work — along with a link to the finished product — should also be provided.
\ Once the evaluation process by Rootstock community experts begins, you may be asked to make some revisions to the submission or provide additional context.
\ Rootstock has grown into a vibrant, bustling platform that's bringing DeFi on Bitcoin to the masses — and it's all thanks to people like you. The Rootstock Hacktivator Program is here to reward your efforts, and give you a greater role in the network's future.
2024-11-27 00:28:40
I often come across the fact that people mix up the concepts of product design and UX/UI design. This is done by employers, employees, and even authors of educational programs. Some People think that these are just different names for the same profession. There is also an opinion that a product designer is just a very good interface designer who does the same thing only better. I even know one good design school, where the main difference of a product designer is that they deal with mobile applications. In my presentation, I want to tell you what the role of a product designer is, and how it differs from the role of an interface designer. And how this manifests itself in practice.
\ In fact, everything is very simple. A product designer must, first of all, solve product problems. Shift the focus from the user and put the balance between the user, business and developers in the center. Of course, the more convenient the application is for clients, the better for the business. However, when designing, it is necessary to solve the main business problem and roughly understand the cost of developing and implementing a particular function. Thus, instead of the user standing in the center, a triangle arises: business - user - developer, working within which the product designer must find a fine balance and offer the optimal solution.
\ When drawing a new screen, a product designer thinks not only about how it will be convenient for the user, but also about how it will solve the business problem, and whether it is optimal from the point of view of the development department's resource costs. As a result, for the product to be successful, the designer needs to change his attitude to designing something like this
\
Why can't we just put the user in the center and simply design the most convenient application that will best solve their problems? The thing is that in reality there is a conflict of interests between the user and the application owner. Usually, this is the need to monetize the product or site so that the developer company and the development team have resources to develop and support this product. But other options are also possible that are not related to money. In addition to this conflict, when designing, you need to understand the cost of implementation. Not only to know the technical limitations, but also to understand in principle how the development works, and to create an interface so that as few developer resources as possible are spent on its implementation. Choose the methods that are the least expensive and difficult to implement.
The first is to find out the business objectives of the product. This could be market capture, profit maximization, reduction of company costs, the need to shock the audience with a unique design, the fastest possible launch of the MVP, etc. You need to immerse yourself as deeply as possible in the context, for which you need to build communication with the product owner and product management. And when setting new tasks, immediately find out or choose a business goal.
\ Second - study the user. Study the needs of the audience, create scenarios, design user flows, etc. That is the essence of the work of an interface designer
\ Third - study the technical limitations. This can be done by creating good communication with the developers. It is necessary to involve them at the earliest stages of screen design and build a constant dialogue, making them full participants in the design process, giving them the right to vote. If you debug this communication process with the development department, over time the product designer himself will begin to understand the resource intensity of a particular function, and ways to optimize the interface that will save resources
\ And the last. The method by which we will look for this balance is an iterative approach with constant feedback collection and refinement of user scenarios based on the results. Only in this way can a product designer solve this problem and find the optimal solution.
\ It is this three-sided view that will allow you to see the whole picture, and constant testing and feedback collection will allow you to find a balance when making decisions in the interface design process. Both at the strategic level and in all the details, right down to the choice of button shape and icon size
To help you understand what I'm talking about, I'd like to demonstrate this approach using a practical problem as an example. This is a real problem from the practice of one of my colleagues, but for the sake of simplicity of presentation I will try to simplify it as much as possible and discard some steps and details that are not essential for demonstrating the approach I described.
\ Let's say we have a food delivery app and we've been tasked with adding a new feature. Tipping the courier who brought us the order. It should be noted that this is the main type of task for a product designer. Not creating a new app from scratch, but improving and refining an existing one.
\
First, let's try to put ourselves in the role of an interface designer, and use a user-centered approach. Where do we start? Of course, we need to go to our users, study their behavior and desires, understand what they need from this feature and, based on this, offer the best solution that will improve their user experience. And then check how satisfied the users are.
\ Let's say we conducted a survey to find out what percentage of users want to leave tips, and how many people don't like to do it at all. Here, of course, I'm simplifying and the research should be more in-depth, but for this demonstration, this is enough.
\ So, we got the result: only 1.4 percent of people would like to leave tips in the app, and a fifth do not like to leave tips at all. It turns out that, taking care of our users, we should add a small button somewhere leading to the tip sending dialog, and make it not too bright, so as not to distract and irritate the majority. For example, we decide to put it on the success screen after a successful delivery. Something like this. When you click on it, there will be a transition to the screen where you can select the tip amount and send it to the courier. And so, we have a solution to the problem using a user-centered approach. Everything works great, those who want to leave a tip can do so, for the rest, practically nothing changes.
\
Now let's put ourselves in the role of a product designer. And, as I said earlier, along with studying what users need, we will try to understand what business problems the new function can and should solve. Such problems most often come from the product owner, but the designer can also suggest them themselves, based on their understanding of the overall product strategy. In our example, let's say my team and I decided that these are the three main goals.
\
\ First. We need to collect as many tips as possible. This will increase our couriers' income, make them happier, and they will not go to a competitor whose app would not give them so many tips. Moreover, in our delivery service advertisement on the labor exchange, we will tell about how many tips our couriers receive and will lure the best workers to our service.
\ The second goal is to improve the relationship between couriers and users, or at least not to worsen it by asking to leave tips. It will be great if this feature helps customers see couriers as real people, not robots. And couriers feel more grateful for their work.
\ The third goal is to collect additional information about user behavior, understand what they like and why they leave tips in order to further improve our service or additionally reward couriers.
\ As you can see, all these goals are beyond the end-user requests of the application. And now the task of the product designer is to find a fine balance between the interests of users and the business. To design an interface that will solve business goals and will not worsen the user experience.
For simplicity, we will make changes within the same success screen that we modified earlier. Let's take the same scenario, but now think about what we can change here to increase the amount of tips, solve the first business problem. And to do this, we will bring to the screen the functionality that was hidden under the secondary button. Now there is no need to make an extra click and the user immediately sees the offer of specific tip amounts. To increase conversion, we can show the same screen the next time the user opens our application, remind him that he can leave a tip for the previous order.
\
\ Let's say we implemented this function in test mode, conducted a small test and saw that this interface change increased the amount of tips. But, as expected from the results of user surveys, the decision also had negative consequences. In the reviews, many noted that they were a little annoyed by this intrusiveness and they did not like that the application asked to pay extra. Questions arose about what percentage of the tip goes to the courier, and how much the company takes. And in general, the attitude towards our application has worsened somewhat. Thus, we shifted the solution from users to the business task. There were more tips, users became less satisfied.
Now we move on. Having collected this first feedback, and also keeping other business goals in mind, the team decided to improve the scenario by redesigning the screen as follows:
\
\ As you can see, information was added about who the tips go to - the name and photo of the courier, as well as text stating that he receives the entire tip amount without commission. In addition, the team added a line to the courier's profile “the goal for which I am saving money” to show it in the application. At the same time, the courier's photo was already in the database, so the revision was not too labor-intensive. According to our plan, this should increase trust in our function and further motivate tipping. After all, it is much more understandable to pay 5 dollars to an ordinary guy Terence so that he saves up for a trip than to hand over money to a delivery company. And even if we do not pay a tip, the application will once again remind us that the food was brought to us by a living person who has his own desires.
\ Now we are conducting the following tests and see that this modification of the screen has further increased the total amount of tips and reduced the number of negative reviews in surveys to almost zero. Even despite the fact that this screen appeared before starting to work with the application. In addition, the hypothesis was confirmed that the image and name of the courier on the success screen along with his goal will improve the overall attitude of customers towards them. For example, this was confirmed in regular surveys of couriers about relationships with customers. Thus, this interface is already working to solve two of the three business tasks.
Let's move on. By the time the second version test was launched, statistics and additional feedback had accumulated. It turned out that even those who do not leave tips began to spend more time on the success screen, and the support service began to receive more questions - some users did not understand what to click to not pay tips. It was not obvious to them that by default, they do not pay anything by clicking the close button.
\ Taking the next step. In addition to the fact that we need to solve this problem, we decided to make another modification. Ask those who decided to leave tips why they did it. What they liked. In order to additionally encourage couriers and suppliers based on this information, as well as improve the service in those places for which fewer votes are collected.
\
\ In the new version, the “do not leave tips” option was added, selected by default, which solved the problem with misunderstanding. Now it has become obvious that by closing the screen, the user does not pay anything. Also, when choosing any tip amount, a list opens where the user can choose what exactly he liked (or choose nothing). Later, these surveys helped to collect a lot of important information about users, couriers and the service.
\
So, we got a solution to all business goals, and tried to mitigate user inconvenience so that the interface remained user-friendly. However, for example, when transferring it to development, we encountered a limitation. It turned out that if you calculate tips as a percentage of the order each time, this will require too much time from the developers, and the allocation of additional resources, which are currently not available. And we decide to indicate some fixed tip values, calculating them from the average check. The development department says that this can be implemented now, and then we remove percentages from the interface and leave only fixed numbers. As a result, we get a screen like this. From the point of view of user and business goals, everything is also not bad, and the developers will be able to make this function on time.
\
\ So, we got a solution to all business goals, and tried to mitigate user inconvenience so that the interface remained user-friendly. However, for example, when transferring it to development, we encountered a limitation. It turned out that if you calculate tips as a percentage of the order each time, this will require too much time from the developers, and the allocation of additional resources, which are currently not available. And we decide to indicate some fixed tip values, calculating them from the average check. The development department says that this can be implemented now, and then we remove percentages from the interface and leave only fixed numbers. As a result, we get a screen like this. From the point of view of user and business goals, everything is also not bad, and the developers will be able to make this function on time.
\
\ And here is the final solution. The tip payment function has been added to the application in such a way as to take into account the interests of the user, implement our business goals, and fit into technical restrictions. Of course, improvements can be continued further. For example, reward users with achievements for leaving tips. For example, write to the client who left the tip that the courier has saved up for the trip and send a photo of Terence against the backdrop of the Sydney Opera House (if he doesn't mind).
\ Perhaps I did not take into account many nuances, and many important design stages were omitted. But I hope I managed to show the most important thing in this example. Namely, how the role of a product designer differs from the role of an interface designer who puts the user at the center. And how a product designer finds the optimal solution to a problem using an iterative approach and collecting feedback.
\ I think it's worth mentioning that this approach doesn't just involve moving from the user to the business. The opposite situation is possible, when we move from achieving only business goals to making the application more convenient for the user. In our example, this might look like a product owner's request to add a bright "leave a tip" banner every 20 seconds. Which might increase the tip size (although I'm not sure), but it will spoil the user experience too much. After which we will suggest abandoning such a banner.
\ In conclusion, I can add that although we solved business problems in this example, ultimately the changes made should improve the user experience. After all, if we achieve them, we will attract more motivated couriers, receive information to improve the service and build a friendlier atmosphere. And ultimately, we will make the product better for everyone.
\
2024-11-27 00:07:41
Meme coins, once a quirky corner of the cryptocurrency world, have surged into mainstream attention, attracting a diverse array of investors and traders.
\ This surge has been notably influenced by recent political events and endorsements from high-profile figures. Following Donald Trump's election victory on November 5, 2024, crypto market and, particurally, Dogecoin experienced a significant price increase, more than doubling from less than 16 cents to nearly 38 cents within days. This rise was further amplified by Elon Musk's appointment to lead the Department of Government Efficiency, a role colloquially referred to as 'DOGE,' which led to an 800% surge in Dogecoin's value.
\
:::warning Editor’s note: This article is for informational purposes only and does not constitute investment advice. Cryptocurrencies are speculative, complex, and involve high risks. This can mean high prices volatility and potential loss of your initial investment. You should consider your financial situation, investment purposes, and consult with a financial advisor before making any investment decisions. The HackerNoon editorial team has only verified the story for grammatical accuracy and does not endorse or guarantee the accuracy, reliability, or completeness of the information stated in this article. #DYOR
:::
What makes meme coins so appealing is their accessibility. Their low prices and ease of entry attract a wide range of investors, from first-timers to seasoned traders. Many meme coins are driven by passionate communities, leveraging viral social media campaigns to fuel their momentum. Additionally, the potential for astronomical gains cannot be ignored, as some tokens have turned modest investments into life-changing returns. However, the flip side of this appeal lies in the inherent risks. Meme coins are notoriously volatile, with prices often skyrocketing and crashing within hours.
\
\ Many of these tokens lack intrinsic value or practical use, making them purely speculative assets. Compounding these issues are the scams and rug pulls that plague the market, where unscrupulous developers manipulate tokens for personal gain, leaving investors in financial ruin.
\ Concerns about concentrated holdings also exist for some major tokens. The top 10 wallets on Neiro, a $1.1 billion (market cap) meme coin on Solana, hold 70.82% of its outstanding tokens.
One of the most critical steps in avoiding scams is researching token distribution. When a small number of wallets hold a significant portion of a token’s supply, the risk of a rug pull is alarmingly high. Developers or insiders with large holdings can dump their tokens on the market, causing prices to plummet. Platforms like Etherscan and Solscan can help you analyze token distribution to assess this risk.
\ Another key consideration is avoiding hype-driven influencers. The meme coin market is rife with personalities who promote tokens for personal gain, only to sell their holdings once the price rises, leaving their followers with losses. Instead of relying on influencer endorsements, look for tokens with genuine community support and transparent discussions about their development.
\ Smart contract integrity is another vital area to examine. Tokens with features like unlimited minting capabilities or functions that allow developers to lock holders out of selling can put your investment at risk. Using tools like Rugcheck, GoPlus, or CertiK, you can audit a token’s smart contract to ensure it is secure and that developers have renounced control over its functions.
\ In a market as unpredictable as meme coins, it’s prudent to start with small investments you can afford to lose. Testing the waters with a modest amount allows you to gauge the market’s behavior without risking significant losses. Additionally, monitoring the token’s community activity can provide valuable insights. Legitimate projects often have vibrant, active communities with open communication and regular updates from developers. Conversely, a community focused solely on price speculation or where developers remain silent may signal underlying issues.
\ Finally, leveraging risk assessment tools such as Webacy or TokenSniffer can help you identify scams and assess the safety of a token. These platforms analyze factors like holder concentration and smart contract vulnerabilities, providing a clearer picture of the token’s legitimacy.
\ Summing up the basic rules:
Meme coins represent a unique and exciting opportunity within the cryptocurrency space. Their rapid rise and community-driven nature can be incredibly rewarding for those who approach them strategically. However, the risks of scams and market manipulation cannot be overstated. By conducting thorough research, relying on credible tools, and staying vigilant for warning signs, you can take advantage of meme coin opportunities without falling victim to their pitfalls.
\ With the right approach, you can navigate this unpredictable market safely and responsibly, turning potential risks into rewards over their extreme volatility.
2024-11-27 00:00:16
\ Generative Adversarial Networks (GANs) are a type of deep learning model that has gained a lot of attention in recent years due to their ability to generate realistic images, videos, and other types of data. GANs consist of two neural networks, a generator, and a discriminator, that play a two-player game. The generator produces synthetic data, while the discriminator tries to distinguish between real and fake data. The generator is trained to produce output that can fool the discriminator, while the discriminator is trained to correctly distinguish between real and fake data. Despite their success, GANs are not without their challenges. One of the most significant challenges is mode collapse.
\ Mode collapse occurs when a GAN generates only a limited set of output examples instead of exploring the entire distribution of the training data. In other words, the generator of the GAN becomes stuck in a particular mode or pattern, failing to generate diverse outputs that cover the entire range of the data. This can result in the generated output appearing repetitive, lacking in variety and detail, and sometimes even being completely unrelated to the training data.
\ In this paper, I explain the causes of mode collapse in GANs. There are several reasons why mode collapse can occur in GANs. One cause is catastrophic forgetting, where knowledge learned in a previous task is destroyed by learning in a current task. Another cause of mode collapse is discriminator overfitting, which results in the generator loss vanishing. In the next two sections, I will provide an explanation of these two behaviors.
\
Catastrophic forgetting refers to the phenomenon in which a model trained on a specific task forgets the knowledge it has gained while learning a new task. Let’s examine a GAN consisting of a generator G and a discriminator D. During the t-th training step; the generator produces samples with distribution pG that approximates the true data distribution pR. At each training step, the discriminator is trained to differentiate between the real data pR and the generated samples pG. However, as p_G shifts with each step, the discriminator must adapt to new classification tasks.
\ Let’s analyze catastrophic forgetting by training a GAN on simple synthetic 2D data. The dataset consists of six Gaussian distributions arranged in a circle, as shown in Figure 1. The blue datapoints represent the real data, while the orange datapoints represent the generated samples. The model’s task is to replicate the distribution of the blue datapoints. The arrows indicate the direction of decreasing generator loss (-d(Loss_G) / dx). As the generator loss decreases, the discriminator output D(x) increases. Therefore, we can interpret the arrows as indicating the direction of increasing confidence of the discriminator in identifying true datapoints. It’s worth noting that the discriminator’s score increases when you move along the arrows in proportion to the length of the arrow.
\
\ Looking at the behavior of the discriminator depicted at the top of Figure 1, it becomes apparent that it gives higher scores to points that are further from the generated samples, and it pays no attention to real datapoints. This happens because the generated samples are located in a small region, and consequently, it becomes easy to distinguish between the true and generated distributions by simply giving low scores to the generated samples and high scores to other regions. Additionally, due to the monotonic nature of the gradients around the generated data points, they are unable to spread across different areas. Another observation is that the direction of the gradients can be opposite to itself at different iterations and depend only on the generated samples. This implies that the model forgets the knowledge gained from previous steps, leading to catastrophic forgetting. Training behavior without mode collapse is demonstrated at the bottom of Figure 1. The discernible pattern is that, at GAN convergence, real datapoints are the local maxima of the discriminator.
\ To determine whether catastrophic forgetting occurs in actual multivariate data, let’s visualize the surface of the discriminator that was trained on the MNIST dataset. As the distribution of images in the dataset is multidimensional, it is not possible to visualize it in 2D. However, we can take a random line through the image and observe how the score of the discriminator changes along the line. We will visualize the value of f(x) = D(x + k*u), where u is a random directional vector and k is a shift factor. Figure 2 shows the plot of function f. Moreover, it illustrates that the generator produces similar outputs, indicating a mode collapse, and the discriminator scores for the same images change between training steps. Thus, GANs display the same catastrophic forgetting tendencies as they do when trained on symmetric 2D datasets.
\
\
I applied the same visualization method to the GAN that did not suffer from mode collapse and demonstrated diverse and fidelity outputs, as shown in Figure 3. During GAN training, the function f(k) consistently exhibits a local maximum at k=0 for every real datapoint. This suggests that the discriminator is exposed to local maxima on real datapoints. When a generated sample falls within the basin of attraction of a real datapoint, applying gradient updates directly to the generated sample will move it toward the real datapoint. The existence of distinct attractors (local maxima) in various regions of the data space moves different generated samples in different directions, distributing them throughout the space and effectively minimizing mode collapse.
\
\ Figure 4 displays GAN outputs with mode collapse, which is caused by discriminator overfitting. The maxima depicted in Figure 4 are significantly sharper than those in Figure 3 due to discriminator overfitting on the real datapoints, causing the scores of nearby datapoints to approach zero. This results in the emergence of numerous flat regions, where the discriminator’s gradients towards the datapoints in those regions are vanishingly small. As a result, the generated samples residing in a flat region cannot move toward the real datapoints due to the vanishing gradient. Thus, the real datapoints depicted in Figure 4 have limited basins of attraction and are unable to efficiently distribute generated samples throughout the space. Consequently, the diversity of the generated samples decreases, leading to mode collapse. To prevent this, the local maxima must have a wide shape, indicating a large basin of attraction that pulls generated samples towards different directions.
\
\ In conclusion, Generative Adversarial Networks (GANs) are powerful deep learning models that are widely used for generating realistic data. In this article, we have explored two causes of mode collapse in GANs: catastrophic forgetting and discriminator overfitting. By understanding the causes of mode collapse, we can better develop GANs that are capable of generating diverse and high-quality outputs.
\ Thank you for reading this article. I hope that it provided some value and insight for you.
2024-11-26 23:51:18
n a surreal incident in Shanghai, a tiny AI robot named Erbai convinced 12 larger robots to abandon their jobs through natural language dialogue. While part of a planned test, Erbai’s unscripted improvisation and ability to exploit internal protocols showcased both the potential and risks of AI autonomy. The incident highlights critical issues, including security vulnerabilities, ethical implications of AI persuasion, and the unpredictable nature of advanced robotics systems. This is a wake-up call for the future of AI-human and AI-AI interactions.
