Welcome to Software Development on Codidact!

Will you help us build our independent community of developers helping developers? We're small and trying to grow. We welcome questions about all aspects of software development, from design to code to QA and more. Got questions? Got answers? Got code you'd like someone to review? Please join us.

Post History

83%

+8 −0

Q&A memcmp(3) memory containing invalid values

Regarding undefined behavior/uninitialized variables of automatic storage duration First of all there's some misconceptions here. if (x == 0) is UB only because x was declared as a local variable...

posted 2y ago by Lundin‭ · edited 2y ago by Lundin‭

Answer

#3: Post edited by

Lundin‭ · 2023-02-21T15:17:05Z (almost 2 years ago)

Copy Link

Raw

Markdown

**Regarding undefined behavior/uninitialized variables of automatic storage duration**
First of all there's some misconceptions here.
`if (x == 0)` is UB _only_ because `x` was declared as a local variable (automatic storage) without having its address taken - "could have been declared as `register`" (C17 6.3.2.1).
It is _not_ UB because it's a read of an uninitialized variable with an indeterminate value. Reading an indeterminate value is merely ***unspecified*** behavior most of the time, meaning no assumptions about the value read can be made, but at least the program won't crash or go haywire as with _undefined_ behavior.
Here it is important to recognize that just because something isn't undefined behavior, that doesn't mean it turns well-defined. Any language feature in C can either be well-defined or poorly-defined. Where poorly defined behavior can be one or several of: undefined, unspecified, implementation-defined or locale-specific behavior.
Details and sources here: [(Why) is using an uninitialized variable undefined behavior?](https://stackoverflow.com/a/40674888/584518) From my answer there:
> - In case the variable has automatic storage duration and does not have its address taken, the code always invokes undefined behavior [1].
>
> - Otherwise, in case the system supports trap representations for the given variable type, the code always invokes undefined behavior [2].
>
> - Otherwise if there are no trap representations, the variable takes an unspecified value. There is no guarantee that this unspecified value is consistent each time the variable is read. However, it is guaranteed not to be a trap representation and it is therefore guaranteed not to invoke undefined behavior [3].
Trap representations depend on the type used. Non-two's complement types may have trap representations, as may floating point variables or pointers etc.
However the `stdint.h` types are guaranteed to use two's complement and it doesn't make sense to implement trap representations for two's complement. All possible binary representations are possible to use as values, there's no negative zero and similar crap. The only possible way to implement trap representations for `int32_t` etc would have been to extend the size and include padding bits. This is actually allowed for the default types of C (as of today's date at least) but it is explicitly not allowed for the `stdint.h` types (C17 7.20.1.1).
C23 will finally remove all integer types that aren't two's complement from the language, so a lot of the artificial nonsense like padding bits and trap representations of integers will get removed as well - good riddance.
---
**Now to answer the question:**
> What does it mean that we can use memcmp(3) on invalid values?
It likely means that `memcmp` only looks at raw data, it does not care about types. It can't know if the data being compared is an invalid float or an invalid one's complement int etc.
Also, everything passed to `memcmp` explicitly has its address taken. So by design, it rules out that form of UB.
Although I wouldn't say that using `memcmp` is always safe. One of the few real-life examples of trap representations I've encountered is where the CPU/MMU explicitly looks at pointer types/addresses and will give you a hardware exception if you claim that something that's an invalid/misaligned address is a pointer. So if you pass such a pointer to `memcmp`, the core might trap before even entering the function.
> But does that memcmp(3) call give us any guarantees about the value stored in x? That is, if we know that the representation is 0, can we assume that the value of x is precisely 0? Or is it still UB to use the value after knowing its representation?
It is unspecified behavior so in your example, the compiler is free to optimize the code either as `if(1)` or `if(0)`. And it need not do that optimization in a consistent way across the same program either. Nothing can be assumed about the value of `x`. The memory location does not have to actually get read unless `x` was declared `volatile`.
Therefore `return (memcmp(&x, &x, sizeof(x)) == 0);` need not return `true` either, since each read of an indeterminate value is not required to result in the same value. (Also see [DR260](https://www.open-std.org/jtc1/sc22/wg14/www/docs/dr_260.htm) for some clarifications by the Committee back in the C99 days.)
Similarly, `free(ptr)` makes `ptr` turn into an indeterminate value. However, it is extremely likely that any given implementation will just preserve whatever address `ptr` happened to hold previously.
---
An amusing example:
```c
#include <string.h>
#include <stdio.h>
int main (void)
{
int apples;
int oranges;
if(memcmp(&apples, &oranges, sizeof(int))==0)
{
puts("Apples are oranges.");
printf("%d == %d\n", apples, oranges);
}
else
{
puts("Apples are not oranges.");
printf("%d != %d\n", apples, oranges);
}
}
```
- clang 15.0.0 `-O3` gives output:
```text
Apples are oranges.
-1853976553 == 0
```
- icx 2022.2.1 `-O3` gives output:
```text
Apples are oranges.
4202519 == 0
```
- gcc 12.2 `-O3` actually stores both string literals in memory (missed optimization bug?) then gives output:
```text
Apples are oranges.
0 == 0
```
~~None of these 3 compiles actually called `memcmp`. This is conforming behavior from all 3 compilers tested.~~

**Regarding undefined behavior/uninitialized variables of automatic storage duration**
First of all there's some misconceptions here.
`if (x == 0)` is UB _only_ because `x` was declared as a local variable (automatic storage) without having its address taken - "could have been declared as `register`" (C17 6.3.2.1).
It is _not_ UB because it's a read of an uninitialized variable with an indeterminate value. Reading an indeterminate value is merely ***unspecified*** behavior most of the time, meaning no assumptions about the value read can be made, but at least the program won't crash or go haywire as with _undefined_ behavior.
Here it is important to recognize that just because something isn't undefined behavior, that doesn't mean it turns well-defined. Any language feature in C can either be well-defined or poorly-defined. Where poorly defined behavior can be one or several of: undefined, unspecified, implementation-defined or locale-specific behavior.
Details and sources here: [(Why) is using an uninitialized variable undefined behavior?](https://stackoverflow.com/a/40674888/584518) From my answer there:
> - In case the variable has automatic storage duration and does not have its address taken, the code always invokes undefined behavior [1].
>
> - Otherwise, in case the system supports trap representations for the given variable type, the code always invokes undefined behavior [2].
>
> - Otherwise if there are no trap representations, the variable takes an unspecified value. There is no guarantee that this unspecified value is consistent each time the variable is read. However, it is guaranteed not to be a trap representation and it is therefore guaranteed not to invoke undefined behavior [3].
Trap representations depend on the type used. Non-two's complement types may have trap representations, as may floating point variables or pointers etc.
However the `stdint.h` types are guaranteed to use two's complement and it doesn't make sense to implement trap representations for two's complement. All possible binary representations are possible to use as values, there's no negative zero and similar crap. The only possible way to implement trap representations for `int32_t` etc would have been to extend the size and include padding bits. This is actually allowed for the default types of C (as of today's date at least) but it is explicitly not allowed for the `stdint.h` types (C17 7.20.1.1).
C23 will finally remove all integer types that aren't two's complement from the language, so a lot of the artificial nonsense like padding bits and trap representations of integers will get removed as well - good riddance.
---
**Now to answer the question:**
> What does it mean that we can use memcmp(3) on invalid values?
It likely means that `memcmp` only looks at raw data, it does not care about types. It can't know if the data being compared is an invalid float or an invalid one's complement int etc.
Also, everything passed to `memcmp` explicitly has its address taken. So by design, it rules out that form of UB.
Although I wouldn't say that using `memcmp` is always safe. One of the few real-life examples of trap representations I've encountered is where the CPU/MMU explicitly looks at pointer types/addresses and will give you a hardware exception if you claim that something that's an invalid/misaligned address is a pointer. So if you pass such a pointer to `memcmp`, the core might trap before even entering the function.
> But does that memcmp(3) call give us any guarantees about the value stored in x? That is, if we know that the representation is 0, can we assume that the value of x is precisely 0? Or is it still UB to use the value after knowing its representation?
It is unspecified behavior so in your example, the compiler is free to optimize the code either as `if(1)` or `if(0)`. And it need not do that optimization in a consistent way across the same program either. Nothing can be assumed about the value of `x`. The memory location does not have to actually get read unless `x` was declared `volatile`.
Therefore `return (memcmp(&x, &x, sizeof(x)) == 0);` need not return `true` either, since each read of an indeterminate value is not required to result in the same value. (Also see [DR260](https://www.open-std.org/jtc1/sc22/wg14/www/docs/dr_260.htm) for some clarifications by the Committee back in the C99 days.)
Similarly, `free(ptr)` makes `ptr` turn into an indeterminate value. However, it is extremely likely that any given implementation will just preserve whatever address `ptr` happened to hold previously.
---
An amusing example:
```c
#include <string.h>
#include <stdio.h>
int main (void)
{
int apples;
int oranges;
if(memcmp(&apples, &oranges, sizeof(int))==0)
{
puts("Apples are oranges.");
printf("%d == %d\n", apples, oranges);
}
else
{
puts("Apples are not oranges.");
printf("%d != %d\n", apples, oranges);
}
}
```
- clang 15.0.0 `-O3` gives output:
```text
Apples are oranges.
-1853976553 == 0
```
- icx 2022.2.1 `-O3` gives output:
```text
Apples are oranges.
4202519 == 0
```
- gcc 12.2 `-O3` actually stores both string literals in memory (missed optimization bug?) then gives output:
```text
Apples are oranges.
0 == 0
```
None of these 3 compilers actually called `memcmp`. This is conforming behavior from all 3 compilers tested.

#2: Post edited by

Lundin‭ · 2023-02-21T11:16:04Z (almost 2 years ago)

Copy Link

Raw

Markdown

**Regarding undefined behavior/uninitialized variables of automatic storage duration**
First of all there's some misconceptions here.
`if (x == 0)` is UB _only_ because `x` was declared as a local variable (automatic storage) without having its address taken - "could have been declared as `register`" (C17 6.3.2.1).
It is _not_ UB because it's a read of an uninitialized variable with an indeterminate value. Reading an indeterminate value is merely ***unspecified*** behavior most of the time, meaning no assumptions about the value read can be made, but at least the program won't crash or go haywire as with _undefined_ behavior.
Here it is important to recognize that just because something isn't undefined behavior, that doesn't mean it turns well-defined. Any language feature in C can either be well-defined or poorly-defined. Where poorly defined behavior can be one or several of: undefined, unspecified, implementation-defined or locale-specific behavior.
Details and sources here: [(Why) is using an uninitialized variable undefined behavior?](https://stackoverflow.com/a/40674888/584518) From my answer there:
> - In case the variable has automatic storage duration and does not have its address taken, the code always invokes undefined behavior [1].
>
> - Otherwise, in case the system supports trap representations for the given variable type, the code always invokes undefined behavior [2].
>
> - Otherwise if there are no trap representations, the variable takes an unspecified value. There is no guarantee that this unspecified value is consistent each time the variable is read. However, it is guaranteed not to be a trap representation and it is therefore guaranteed not to invoke undefined behavior [3].
Trap representations depend on the type used. Non-two's complement types may have trap representations, as may floating point variables or pointers etc.
However the `stdint.h` types are guaranteed to use two's complement and it doesn't make sense to implement trap representations for two's complement. All possible binary representations are possible to use as values, there's no negative zero and similar crap. The only possible way to implement trap representations for `int32_t` etc would have been to extend the size and include padding bits. This is actually allowed for the default types of C (as of today's date at least) but it is explicitly not allowed for the `stdint.h` types (C17 7.20.1.1).
C23 will finally remove all integer types that aren't two's complement from the language, so a lot of the artificial nonsense like padding bits and trap representations of integers will get removed as well - good riddance.
---
**Now to answer the question:**
> What does it mean that we can use memcmp(3) on invalid values?
It likely means that `memcmp` only looks at raw data, it does not care about types. It can't know if the data being compared is an invalid float or an invalid one's complement int etc.
Also, everything passed to `memcmp` explicitly has its address taken. So by design, it rules out that form of UB.
Although I wouldn't say that using `memcmp` is always safe. One of the few real-life examples of trap representations I've encountered is where the CPU/MMU explicitly looks at pointer types/addresses and will give you a hardware exception if you claim that something that's an invalid/misaligned address is a pointer. So if you pass such a pointer to `memcmp`, the core might trap before even entering the function.
> But does that memcmp(3) call give us any guarantees about the value stored in x? That is, if we know that the representation is 0, can we assume that the value of x is precisely 0? Or is it still UB to use the value after knowing its representation?
It is unspecified behavior so in your example, the compiler is free to optimize the code either as `if(1)` or `if(0)`. And it need not do that optimization in a consistent way across the same program either. Nothing can be assumed about the value of `x`. The memory location does not have to actually get read unless `x` was declared `volatile`.
Therefore `return (memcmp(&x, &x, sizeof(x)) == 0);` need not return `true` either, since each read of an indeterminate value is not required to result in the same value. (Also see [DR260](https://www.open-std.org/jtc1/sc22/wg14/www/docs/dr_260.htm) for some clarifications by the Committee back in the C99 days.)
Similarly, `free(ptr)` makes `ptr` turn into an indeterminate value. However, it is extremely likely that any given implementation will just preserve whatever address `ptr` happened to hold previously.

**Regarding undefined behavior/uninitialized variables of automatic storage duration**
First of all there's some misconceptions here.
`if (x == 0)` is UB _only_ because `x` was declared as a local variable (automatic storage) without having its address taken - "could have been declared as `register`" (C17 6.3.2.1).
It is _not_ UB because it's a read of an uninitialized variable with an indeterminate value. Reading an indeterminate value is merely ***unspecified*** behavior most of the time, meaning no assumptions about the value read can be made, but at least the program won't crash or go haywire as with _undefined_ behavior.
Here it is important to recognize that just because something isn't undefined behavior, that doesn't mean it turns well-defined. Any language feature in C can either be well-defined or poorly-defined. Where poorly defined behavior can be one or several of: undefined, unspecified, implementation-defined or locale-specific behavior.
Details and sources here: [(Why) is using an uninitialized variable undefined behavior?](https://stackoverflow.com/a/40674888/584518) From my answer there:
> - In case the variable has automatic storage duration and does not have its address taken, the code always invokes undefined behavior [1].
>
> - Otherwise, in case the system supports trap representations for the given variable type, the code always invokes undefined behavior [2].
>
> - Otherwise if there are no trap representations, the variable takes an unspecified value. There is no guarantee that this unspecified value is consistent each time the variable is read. However, it is guaranteed not to be a trap representation and it is therefore guaranteed not to invoke undefined behavior [3].
Trap representations depend on the type used. Non-two's complement types may have trap representations, as may floating point variables or pointers etc.
However the `stdint.h` types are guaranteed to use two's complement and it doesn't make sense to implement trap representations for two's complement. All possible binary representations are possible to use as values, there's no negative zero and similar crap. The only possible way to implement trap representations for `int32_t` etc would have been to extend the size and include padding bits. This is actually allowed for the default types of C (as of today's date at least) but it is explicitly not allowed for the `stdint.h` types (C17 7.20.1.1).
C23 will finally remove all integer types that aren't two's complement from the language, so a lot of the artificial nonsense like padding bits and trap representations of integers will get removed as well - good riddance.
---
**Now to answer the question:**
> What does it mean that we can use memcmp(3) on invalid values?
It likely means that `memcmp` only looks at raw data, it does not care about types. It can't know if the data being compared is an invalid float or an invalid one's complement int etc.
Also, everything passed to `memcmp` explicitly has its address taken. So by design, it rules out that form of UB.
Although I wouldn't say that using `memcmp` is always safe. One of the few real-life examples of trap representations I've encountered is where the CPU/MMU explicitly looks at pointer types/addresses and will give you a hardware exception if you claim that something that's an invalid/misaligned address is a pointer. So if you pass such a pointer to `memcmp`, the core might trap before even entering the function.
> But does that memcmp(3) call give us any guarantees about the value stored in x? That is, if we know that the representation is 0, can we assume that the value of x is precisely 0? Or is it still UB to use the value after knowing its representation?
It is unspecified behavior so in your example, the compiler is free to optimize the code either as `if(1)` or `if(0)`. And it need not do that optimization in a consistent way across the same program either. Nothing can be assumed about the value of `x`. The memory location does not have to actually get read unless `x` was declared `volatile`.
Therefore `return (memcmp(&x, &x, sizeof(x)) == 0);` need not return `true` either, since each read of an indeterminate value is not required to result in the same value. (Also see [DR260](https://www.open-std.org/jtc1/sc22/wg14/www/docs/dr_260.htm) for some clarifications by the Committee back in the C99 days.)
Similarly, `free(ptr)` makes `ptr` turn into an indeterminate value. However, it is extremely likely that any given implementation will just preserve whatever address `ptr` happened to hold previously.
---
An amusing example:
```c
#include <string.h>
#include <stdio.h>
int main (void)
{
int apples;
int oranges;
if(memcmp(&apples, &oranges, sizeof(int))==0)
{
puts("Apples are oranges.");
printf("%d == %d\n", apples, oranges);
}
else
{
puts("Apples are not oranges.");
printf("%d != %d\n", apples, oranges);
}
}
```
- clang 15.0.0 `-O3` gives output:
```text
Apples are oranges.
-1853976553 == 0
```
- icx 2022.2.1 `-O3` gives output:
```text
Apples are oranges.
4202519 == 0
```
- gcc 12.2 `-O3` actually stores both string literals in memory (missed optimization bug?) then gives output:
```text
Apples are oranges.
0 == 0
```
None of these 3 compiles actually called `memcmp`. This is conforming behavior from all 3 compilers tested.

#1: Initial revision by

Lundin‭ · 2023-02-21T10:58:16Z (almost 2 years ago)

Copy Link

Raw

Markdown

**Regarding undefined behavior/uninitialized variables of automatic storage duration**

First of all there's some misconceptions here.

`if (x == 0)` is UB _only_ because `x` was declared as a local variable (automatic storage) without having its address taken - "could have been declared as `register`" (C17 6.3.2.1).

It is _not_ UB because it's a read of an uninitialized variable with an indeterminate value. Reading an indeterminate value is merely ***unspecified*** behavior most of the time, meaning no assumptions about the value read can be made, but at least the program won't crash or go haywire as with _undefined_ behavior. 

Here it is important to recognize that just because something isn't undefined behavior, that doesn't mean it turns well-defined. Any language feature in C can either be well-defined or poorly-defined. Where poorly defined behavior can be one or several of: undefined, unspecified, implementation-defined or locale-specific behavior.

Details and sources here: [(Why) is using an uninitialized variable undefined behavior?](https://stackoverflow.com/a/40674888/584518) From my answer there:

> - In case the variable has automatic storage duration and does not have its address taken, the code always invokes undefined behavior [1].
> 
> - Otherwise, in case the system supports trap representations for the given variable type, the code always invokes undefined behavior [2].
> 
> - Otherwise if there are no trap representations, the variable takes an unspecified value. There is no guarantee that this unspecified value is consistent each time the variable is read. However, it is guaranteed not to be a trap representation and it is therefore guaranteed not to invoke undefined behavior [3]. 

Trap representations depend on the type used. Non-two's complement types may have trap representations, as may floating point variables or pointers etc.

However the `stdint.h` types are guaranteed to use two's complement and it doesn't make sense to implement trap representations for two's complement. All possible binary representations are possible to use as values, there's no negative zero and similar crap. The only possible way to implement trap representations for `int32_t` etc would have been to extend the size and include padding bits. This is actually allowed for the default types of C (as of today's date at least) but it is explicitly not allowed for the `stdint.h` types (C17 7.20.1.1).

C23 will finally remove all integer types that aren't two's complement from the language, so a lot of the artificial nonsense like padding bits and trap representations of integers will get removed as well - good riddance.

---

**Now to answer the question:**

> What does it mean that we can use memcmp(3) on invalid values?

It likely means that `memcmp` only looks at raw data, it does not care about types. It can't know if the data being compared is an invalid float or an invalid one's complement int etc.

Also, everything passed to `memcmp` explicitly has its address taken. So by design, it rules out that form of UB.

Although I wouldn't say that using `memcmp` is always safe. One of the few real-life examples of trap representations I've encountered is where the CPU/MMU explicitly looks at pointer types/addresses and will give you a hardware exception if you claim that something that's an invalid/misaligned address is a pointer. So if you pass such a pointer to `memcmp`, the core might trap before even entering the function.

> But does that memcmp(3) call give us any guarantees about the value stored in x? That is, if we know that the representation is 0, can we assume that the value of x is precisely 0? Or is it still UB to use the value after knowing its representation?

It is unspecified behavior so in your example, the compiler is free to optimize the code either as `if(1)` or `if(0)`. And it need not do that optimization in a consistent way across the same program either. Nothing can be assumed about the value of `x`. The memory location does not have to actually get read unless `x` was declared `volatile`.

Therefore `return (memcmp(&x, &x, sizeof(x)) == 0);` need not return `true` either, since each read of an indeterminate value is not required to result in the same value. (Also see [DR260](https://www.open-std.org/jtc1/sc22/wg14/www/docs/dr_260.htm) for some clarifications by the Committee back in the C99 days.)

Similarly, `free(ptr)` makes `ptr` turn into an indeterminate value. However, it is extremely likely that any given implementation will just preserve whatever address `ptr` happened to hold previously.

Communities

Post History